Geometric Algebra Planes: Convex Implicit Neural Volumes
Authors: Irmak Sivgin
Sara Fridovich-Keil
Gordon Wetzstein
Mert Pilanci
Venue: ICML 2025 poster (2025)
Keywords: Volume representation
tensor decomposition
convex optimization
geometric algebra
nerf
Primary area: optimization->convex

Abstract: Volume parameterizations abound in recent literature, encompassing methods from classic voxel grids to implicit neural representations. 
While implicit representations offer impressive capacity and improved memory efficiency compared to voxel grids, they traditionally require training through nonconvex optimization, which can be slow and sensitive to initialization and hyperparameters. 
We introduce GA-Planes, a novel family of implicit neural volume representations inspired by Geometric Algebra that can be trained using convex optimization, addressing the limitations of nonconvex methods. 
GA-Planes models generalize many existing representations including any combination of features stored in tensor basis elements followed by a neural feature decoder, and can be adapted to convex or nonconvex training as needed for various inverse problems. 
In the 2D setting, we prove GA-Planes models are equivalent to a low-rank plus low-resolution matrix factorization that outperforms the classic low-rank plus sparse decomposition for fitting a natural image. 
In 3D, GA-Planes models exhibit competitive expressiveness, model size, and optimizability across tasks such as radiance field reconstruction, 3D segmentation, and video segmentation.

Geometric Algebra Planes: Convex Implicit Neural Volumes
Irmak Sivgin 1
Sara Fridovich-Keil 12 Gordon Wetzstein Mert Pilanci
Abstract
ies. In recent years tremendous progress has been made in modeling these volumes using measurements and com- putation (Tewari et al., 2022), to make them accessible for downstream tasks in applications including manufacturing (Intwala & Magikar, 2016; Šlapak et al., 2024), robotic navigation (Ming et al., 2024; Wijayathunga al., 2023), entertainment and culture (Liu et al., 2024; Croce al., 2023), and medicine (Udupa & Herman, 1999; Masero et al., 2002; Richter et al., 2024; Xu et al., 2024). All methods that seek to model volume face three-way tradcoff between model size, which determines hardware memory require- ments, expressiveness, which determines how faithfully the model can represent the underlying volume, and optimiz- ability, which captures how quickly and reliably the model can learn the volume from measurements. Certain appli- cations place stricter requirements on model size (e.g. for deployment on mobile or edge devices), expressiveness (c.g. resolution required for medical diagnosis or safe robotic navigation), or optimizability (e.g. for interactive applica- tions), but all stand to benefit from improvements this three-way pareto frontier.
Volume parameterizations abound in recent liter- ature, encompassing methods from classic voxel grids to implicit neural representations. While implicit representations offer impressive capac- ity and improved memory efficiency compared to voxel grids, they traditionally require training through nonconvex optimization, which can be slow and sensitive to initialization and hyperpa- rameters. We introduce GA-Planes, a novel fam- ily of implicit neural volume representations in- spired by Geometric Algebra that can be trained using convex optimization, addressing the limi- tations of nonconvex methods. GA-Planes mod- els generalize many existing representations in- cluding any combination of features stored in tensor basis elements followed by a neural fea- ture decoder, and can be adapted to convex or nonconvex training as needed for various inverse problems. In the 2D setting, we prove GA- Planes models are equivalent to a low-rank plus low-resolution matrix factorization that outper- forms the classic low-rank plus sparse decom- position for fitting a natural image. In 3D, GA- Planes models exhibit competitive expressiveness, model size, and optimizability across tasks such radiance field reconstruction, 3D segmenta- tion, and video segmentation. Code is available at https: http://gtthhhhch///ooo / viini rmak/ Geometric iepeetrrccAggbbaaaa waw.ggabaaaaaaass.....o
Many existing strategies have been successfully applied at different points along this pareto frontier; some represen- tative examples from computer vision are summarized in Appendix A.1. Our goal is to maintain or surpass the exist- ing pareto frontier of model size and expressiveness while improving optimization stability through convexity.
Our approach introduces conver and semiconvex reformu- lations of the volume modeling optimization process that apply to a broad class of volume models we call Geometric Algebru Planes, or GA-Planes for short. We adopt the term semiconvex for Burer-Monteiro (BM) factorizations of a convex objective, as introduced in (Sahiner et al., 2024), within the context of convex neural networks. BM factor- ized problems have the property that every local minimum is globally optimal (Sahiner et al., 2024).
1. Introduction
Volumes are everywhere from the world we live in to the videos we watch to the organs and tissues inside our bod-
'Equal contribution 'Department of Electrical Engineering, Stanford University, CA, USA "School of Electrical & Computer Engineering, Georgia Institute Technology, GA, USA. Corre- spondence to: Mert Pilanci <pilanci stanford.edu>.
Glanna is a mf primmtives pfrmmive-o-prrmiiivvss model that generalizes several existing volume models including voxels and tensor factorizations. Most importantly, most models in this family can formulated for optimization by a convex program, as long as the objective function (to fit measurements of the volume) is convex. At the same time, any GA-Pland model
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Geometric Algebra Planes
can also be formulated for nonconvex optimization towards any objective, matching the range of applicability enjoyed by common models. While only our convex and semicon- vex models come with guarantees of convergence to global optimality, all the models we introduce extend the pareto frontier of model size, expressiveness, and optimizability on diverse tasks.
Yu al., 2022) and Direct Voxel Grid Optimization (Sun et al., 2022) can optimize quickly but require large model size to achieve good expressiveness (resolution). Many other methods (Chen et al., 2022; Fridovich- et al., 2023; Müller et al., 2022; Kerbl et al., 2023; Reiser et al., 2023; Lombardi et al., 2021) find their niche somewhere in be- tween, achieving tractable model size, good expressiveness, and reasonably fast optimization time exchange for some increased sensitivity (to initialization, randomness, and prior knowledge) in the optimization process. GA-Planes matches or exceeds the performance of strong baselines (Chen et al., 2022; Fridovich- et al., 2023; Barron et al., 2021) in terms of model size and expressiveness, while introducing the option to train by convex or semiconvex optimization with guaranteed convergence to global optimality.
Concretely, we make the following contributions:
We introduce GA-Planes, a mixture-of prrmmmttives vol- ume parameterization inspired by geometric algebra basis elements. GA-Planes combines any subset of line, plane, and volume features at different resolu- tions, with an MLP decoder. This GA-Planes family of parameterizations generalizes many existing volume and radiance field models.
Radiance field modeling. Most of the works described above are designed for radiance field modeling, in which the training measurements consist of color photographs from known camera poses. The goal is then to faithfully model the optical density and view-dependent color of light inside volume so that unseen views can be rendered accurately. A Although we do demonstrate superior performance of GA- Planes in this setting, we note that the volumetric rendering formula used in radiance field modeling (Max, 1995; Kajiya, 1986; Mildenhall al., 2020) yields a nonconvex photomet- ric loss function, regardless of model parameterization.
We derive convex and semiconvex reformulations of the GA-Planes training process for certain tasks and large subset of the GA-Planes model family, to ensure our model optimizes globally regardless of initializa- tion.
We analyze GA-Planes in the 2D setting and show equivalence to low-rank plus low-resolution matrix approximation whose expressiveness can be directly controlled by design choices. We demonstrate that this matrix dccomposition is expressive for natural images, outperforming the classic low-rank plus sparse approximation.
3D segmentation. We test our convex and semiconvex GA-Planes parameterizations on fully convex objectives, namely volume (xyz) segmentation with either indirect 2D tomographic supervision or direct 3D supervision, as well as video (xyt) segmentation. This 3D (xyz) segmentation task has also been studied in recent work (Cen et al., 2023; Uy et al., 2023), though these methods require additional inputs such as a pretrained radiance field model or monocular depth estimator. Our setup is most similar to (Mescheder et al., 2019), which uses implicit neural representation trained with cross-entropy loss and direct 3D supervision of the occupancy function. Instead of having direct access to this 3D training data, we infer 3D supervision labels via Space Carving (Kutulakos & Scitz, 1999) from 2D image masks obtained by image segmentation (via (Kirillov et al., 2023)).
We demonstrate convex, semiconvex, and nonconvex GA-Planes' high performance in terms of memory, ex- pressiveness, and optimizability across three volume- fitting tasks: 3D radiance field reconstruction, 3D seg- mentation, and video segmentation.
2. Related Work
Volume parameterization. Many volume parameteriza- tions have been proposed and enjoy widespread use in di- verse applications. Here we give overview of repre- sentative methods used in computer vision, focusing on methods that parameterize an entire volume (rather than e.g. a surface). These parameterizations exhibit different tradeoffs between memory usage, representation quality, and ease of optimization; richer deseriptions are provided in Appendix A.
Convex neural networks. Recent work has exposed an equivalence between training a shallow (Pilanci & Ergen, 2020) or deep (Ergen & Pilanci, 2024) neural network and solving a convex program whose structure is defined by the architecture and parameter dimensions of the correspond- ing neural network. The key idea behind this procedure is to enumerate (or randomly sample from) the possible activation paths through the neural network, and then treat these paths as a fixed dictionary whose coefficients may be optimized according to a convex program.
Coordinate MLPs like NeRF (Mildenhall et al., 2020) and Scene Representation Networks (Sitzmann et al., 2019b) are representative of Implicit Neural Representations (INRs), which excel at reducing model size (with decent expressive- ness) but suffer from slow optimization. Explicit voxel grid representations like Plenoxels (Sara Fridovich- and Alex
2

---

Geometric Algebra Planes
Geometric (Clifford) Algebra. Our model is inspired by Geometric Algebra (GA), which is powerful framework for modeling geometric primitives (Dorst et al., 2009). The fundamental entity in GA is the multivector, which is sum of vectors, bivectors, trivectors, etc. In 3DD GA, an example is the multivector er + ejeze,, representing the sum of a bivector (a plane) and trivector (a volume). The geometric product in GA allows us to derive a volume element by multiplying plane and line, e.g. (er02)e3 e3e2c3. We use the shorthand 2123- = e2e2. and similarly for other multivector components throughout. We define the GA- Planes model family to include any volume parameterization that combines any subset (including the complete subset and the empty subset) of the linear geometric primitives {el, e2, e3}. planar geometric primitives {e12, e13, @23). and/or volumetric primitive (e123} into a trivector, with (potentially convexified) MLP feature decoder. While our approach leverages the linear algebraic foundations of GA to construct feature grids, it does not fully exploit the entire range of GA operations and avoids the computationally heavy machinery typically associated with GA. Previous works have integrated GA multiplications and equivariance properties into NN layers (Brandstetter et al., 2022; Ruhe et al., 2023a;b), however, this work is the first to use GA in neural volume models to our knowledge.
where Te projects y onto the coordinates of the c'th feature grid ge and db denotes (bi/tri)linear interpolation. The re- sulting feature e, is vector of length di if e E (1,,,3). dg if e E {12, 13, 23} or d3 if c = 123. We repeat this projection and interpolation procedure over each ge. and combine the resulting feature vectors by any combination of elementwise multiplication (o), addition (+), and con- catenation (o) along the feature dimension. Finally, the combined feature vector is decoded using MLP decoder D. The decoder can take as input both the feature vector arising from the feature grids as well as possible auxiliary inputs, such as (positionally encoded) viewing direction. We consider any model that fits the above description to fall into the GA-Planes family. The specific models we use for nonconvex, semiconvex, and convex optimization are illustrated in Figure 1.
Our experiments focus primarily on two specific GA-Planes models that exemplify some of the strongest convex and nonconvex representations in the GA-Planes family. For our experiments including convex optimization, namely 3D segmentation with 2D or 3D supervision, and video segmen- tation, we use the following GA-Planes model (illustrated in the second and third rows of Figure 1) which can be trained by either convex, semiconvex, or nonconvex optimization as described in the following subsections:
3. Model
D(e1 02
C12 C C13 C23 C 0123).
(2)
3.1. The GA-Planes Model Family
Here we use @ to denote concatenation of features. For our radiance field experiments, since the objective function is inherently nonconvex, we use the following nonconvex member of the GA-Planes family (illustrated with multires- olution feature grids in the first row of Figure l):
A GA-Planes model represents a volume using a combina- tion of geometric algebra features e, derived by interpolat- ing the following parameter grids:
Line (I-dimensional) feature grids (gi, g2, ga), where each grid has shape [ri, di] with spatial resolution and feature dimension d1.
O (e) 0e2s o (22 (1ece)))) 11123)
Plane (2-dimensional) feature grids {g12, g13, g2s }, where each grid has shape r2, r2, de] with spatial res- olution r3 and feature dimension d2.
which leverages geometric algebra to multiply (o) lower- dimensional (vector and bivector) features together into 3D volume (trivector) features, but cannot be convexified be- cause of this multiplication. We use multi-resolution copies of the line and plane feature grids gi, g2, 1a.g3, g12, g13; g23, but only a single resolution for the volume grid g123 since it is already at lower resolution. For our nonconvex exper- iments the decoder D is standard fully-connected ReLU neural network; decoder details for our semiconvex and convex models are presented in the following subsections.
A single volume feature grid {giaa} with shape [Ta, T3, T3.77, dz].
A GA-Planes model may include copies of given basis element with different resolution and feature dimensions, to effectively capture multi-resolution signal content. The x, y, and z spatial resolutions of each grid may also dif- fer in practice; for notational simplicity we write isotropic resolutions.
3.2. Semiconvex GA-Planes
For our segmentation experiments (with volumes and vidcos), we use the GA-Planes architecture in Equation (2), with concatenation instead of multiplication of features; we denote this concatenated feature vector as f()) the input to the decoder. Our semiconvex formulation of this model
We first extract features corresponding to = (x, y, 2) E 3 from each of our line. plane, and volume feature grids ge by linear, bilinear, and trilinear interpolation, respectively:
C₂ := (g ne (g))
3

---

Geometric Algebra Planes
MLP
6
MLP
>
Figure 1: Overview of the GA-Planes models we use in our experiments. Our nonconvex model (top) uses a standard MLP decoder and multiplication features when the result yields a volume under geometric algebra; it also concatenates features across multi-resolution grids. Our semiconvex (middle) and convex (bottom) models use a single resolution grid, and avoid multiplication of features since that would induce nonconvexity. The pastel-colored grids inside the indicator function of the convex model are frozen at initialization and used fixed ReLU gating patterns. O denotes concatenation and denotes elementwise multiplication.
each feature
uses convex MLP (Pilanci & Ergen, 2020) as the decoder:
where the features are interpolated from optimizable parameter grids with feature dimension d(c) E {di, d2, da), whereas the gating variables e, inside the indicator function are derived from the same grids frozen at their initialization values to preserve convexity. Here denotes elementwise product of vectors. These indicator functions take the same role the ReLU in nonconvex MLP, using sampling of random activation patterns based on the grid values at initialization.
h ((q) Σ (w, W. f(a)= 2 0].
(4)
Here denotes the trainable hidden layer MLP weights, and W denotes the same weights frozen at initialization in- side the indicator function. The indicator function, denoted I [*). returns 1 if the argument is true, and O otherwise. The indicator function here serves random gating pattern that takes the place of the ReLU in standard nonconvex MLP, where the gating pattern would be optimized rather than fixed in a random pattern. Although this MLP decoder fully convex, we refer to this model as semiconvex (in particular biconvex; see (Sahiner et al., 2024)) because the combined grid features (() are multiplied by the train- able MLP hidden layer weights W, though the objective is separately convex in each of these parameters.
4. Theory
4.1. Equivalence Matrix Completion in 2D
In three dimensions, the complete set of geometric algebra feature grids are those that we include in the GA-Planes fam- ily: {g1, B2: B3, B12- 813- B23, 8123)- In two dimensions, this reduces to: (gi, g2, g12) In this 2D setting, can alyze different members of our GA-Planes family and show equivalence to various formulations of the classic matrix completion problem.
3.3. Convex GA-Planes
For our segmentation experiments (with volumes and videos), we also present fully convex GA-Planes model that is similar to the semiconvex model described above,
Notation. As usual, we use o to denote elementwise mul- tiplication and O to denote concatenation. We use 11xb except that we fuse the learnable weights the MLP de-
to denote the all-ones matrix of size and [|]] de- coder with the weights of the feature mapping, to remove the product of parameters (which is semiconvex but not convex). Our convex model is:
note the indicator function, which evaluates when its argument is positive and 0 otherwise. Our theorem state- ments consider equivalence to matrix completion prob- lem with target matrix M E mm and low-rank compo- nents U E R V E R to be optimized. We include theoretical results for 2D GA-Planes models that combine features by addition (+), multiplication (o), or concatena-
(())= =
Σ
(e o e. 2 0]), (5)
c11..23312213322..22

---

Geometric Algebra Planes
tion (o) and decode features using linear decoder (as warmup); convex MI.,, or nonconvex MLP. The most illuminating results are presented in the theorem statements that follow; the rest (and all proofs) are deferred to Ap- pendix A.2.
right singular vectors and the decoder a learns the singular values of M. This is the optimal rank approximation of matrix M.
Theorem 2. The two-dimensional representation
D(ei + e2 + e12) with linear decoder D(f(g)) - a² f(a)
is equivalent to a low-rank plus low-resolution matrix completion model with the following structure:
Assumptions. Our theorem statements assume that the line feature grids have the same spatial resolution as the target matrix, and thus do not specify the type of interpola- tion. However, the results hold even if the dimensions do not match and nearest neighbor interpolation is used; the empirical performance is similar even slightly improved in practice by using (bi)linear interpolation of features (see Appendix A.3 and A.2 for discussion of other interpola- tion methods). The theorems assume that the optimization objective is to minimize the Frobenius norm of the error matrix; this is equivalent to minimizing mean squared error measured directly in the representation space. This objec- tive function is the one we use for our convex experiments (video and volume segmentation fitting), where we have access to direct supervision; our radiance field experiments instead use indirect measurements (along rays) that are not equivalent to the setting of the theorems.
min | (U'Ikx + LmxkVT + (())) (8) U.V.I.
where L E RmI is the low-resolution component to be learned, with upsampling (interpolation) function s. These two models are equivalent in the sense that U gi diag(a")) V = gīdiag("") (a)aag(a), and L' = =iza", where U V I.* the optimal solution to the low-rank plus low- resolution matrix completion problem in Equation (8) and si, 82, gi2: a" are the optimal grid features and linear de- coder for the D(e e2 + e12) model.
The two noo-dimessional representation D(e1 o C2 + e12) with the same linear decoder is equivalent to a low-rank plus low-resolution matrix completion model:
min U.V.L
II - (UV" + (T.)))
(9)
Theorem 1. The two-dimensional representation
D(e) e2) with linear decoder D(f(g)) = f4) is equivalent to a low-rank matrix completion model with the following structure:
These two models are equivalent in the same sense as above, except that V==g;;
Remark. Theorem describes the behavior of a 2D,
min
M| - (U1xxn + Lmxk )|e U.V
(6)
linear-decoder version of our GA-Planes model, both the ver- sion with addition/cooncatenatio of features (Equation (2)) and the version with multiplication of features (Equa- tion (3)). Extending the same idea to 3D, we can interpret GA-Planes as low-rank plus low-resolution approximation of 3D tensor (volume). We can understand this model as first fitting a low-resolution volume and then finding low-rank approximation to the high-frequency residual volume. When we use multiplication of features, the low- rank residual approximation is optimal and analogous to the rank-thresholded SVD.
These two models are equivalent in the sense that U gi diag(a") and V = gzdiag(a") where U V* is the optimal solution to the low-rank matrix completion problem in Equation (6) and gi, g2, a" are the optimal features and linear decoder for the D(e + +e2) model.
The two-dimensional representation D(e o e2) with the same linear decoder is equivalent to the standard low-rank matrix completion model:
Theorem 3. The two-dimensional representation D(c1 o C2) with a two-layer convex MLP decoder D(f(1) Eln(W;; [()) f(q) ス 0] is
min |M - 1V U.V
These two models are equivalent in the same sense as above, except that V == =g-
equivalent a masked low-rank matrix completion model:
min U.V.W
M
WU, o B
(10)
Using a linear decoder reveals a dramatic differ- ence in representation capacity between feature addition (or concatenation) and multiplication. Using addition, the maxi- mum rank of the matrix approximation is 2 regardless of the feature dimension k. Using multiplication, the maximum rank of the approximation is k. With feature multiplication, the optimal values of the feature grids are identical to the thresholedddd singular value decomposition (SVD) of M, where the feature grids gi and g2 recover the left and
Remark.
where W E kgic contains the trainable weights of the convex MLP decoder, with indices j j= 1,....,K for the input dimension and i == 1,...., .h for the hidden layer dimension. B, E Immxn denotes the binary masking matrix formed by random, fixed gates of the convex MLP decoder; B, = 1 E W 2 Oj. where W denotes the weight matrix W with values fixed at random initialization.
5

---

Geometric Algebra Planes
This mairix completion model and our GA-Planes model D(e o 02) with convex MLPP decoder are equivalent in the sense that U* = gi. V E2- and W W*, where "*, W is the optimal solution to the masked low-rank matrix completion problem Equation (10) and gì B2. W* are the optimal grid features and convex MLP decoder weights for the D(e) o e2) model. The optimal mask matri- ces Bi are defined by the fixed random weight initialization W and the optimal singular vector matrices U V*
Model
Linear decoder
convex MLP decoder
MLP decoder
D(e + ez)
2
F1
V1
D[e @ ea)
2
11
D(e e2)
A
Table 1: Maximum attainable ranks of different 2D GA- Planes models, using only line features. Here l is the feature dimension and ri is the spatial dimension of the features, which need never exceed min(m, n). Replacing a linear decoder with a convex or nonconvex MLP can dramatically increase the rank of the representation.
Remark. We can interpret the matrix completion model of Equation (10) as a sum of different low-rank approxi- mations, where the matrices within each of the h groups are constrained share the same singular vectors U, Vz. The binary masks B, effectively allow each of these h low-rank approximations to attend to (or complete) a different part of the matrix M before being linearly combined through the weights (singular values) W... The upper limit of the rank of this matrix approximation is thus min(n, m)) because the mask matrices can arbitrarily increase the rank beyond the constraint faced by models with a linear decoder. Note that if the feature grids gı and ge have spatial resolution 71 less than min(n, (n. m). the maximum rank will be rı-
4.2. Practical Implications
The upper bounds on model expressivity summarized in Table inform practitioners that (1) If the decoder is linear (or more generally, simple), features should be combined by multiplication rather than addition or concatenation, and (2) If an MLP decoder is used, the rank of the representa- tion will be limited by the resolution of the grid features rather than by their feature dimension. Indeed this second theoretical insight is practically useful in finding the set of parameter allocations that work best in radiance field experiments (see Figure 3 and Table 5), in which we priori- tize maintaining some features with high resolution even at the cost of reducing the feature dimension.
Theorem 4. The two-dimensional representation D(c1 02) with standard two-layer MLP decoder D(((4)) = af (W [(q))+ is equivalent to a low-rank matrix completion model with the following structure:
The upper bounds in Table are also complemented by 2D image fitting experiments in Figure 7, which directly inform practice in 3 ways beyond the insights from Table 1: (1) combining features by multiplication rather than addition is beneficial even when using an MLP decoder, (2) a stan- dard nonconvex MI. decoder outperforms a convex MLP decoder, even though both representations have the same maximum rank, and (3) continuous interpolation of features outperforms nearest neighbor interpolation of features.
h
min U.V.W.a
M
Wo, Σ
(11)
where W E Biik is the weight matrix for the MLP de- coder's hidden layer (with width h) and a E Rh is the weight vector of the MLP decoder's 's output layer.
This matrix completion model and our GA-Planes model D(e) e2) with nonconvex MLP decoder are equivalent in the sense that U = gi, V*= g2, W = W', and a o", where U*,V* W a is the optimal solution to the masked low-rank matrix completion problem Equation (11) and gi, 82, W*, xx are the optimal grid features and MLP decoder for the D(e a e2) model.
4.3. Interpretation: Low Rank + Low Resolution
Combining multiple parameterization strategies with com- plementary representation capacities is a time-honored strat- egy in signal processing. A classic example is the combina- tion of sparse and low-rank models used to represent matri- ces in the compressive sensing literature (Chandrasekaran et al., 2009). As shown in Theorem 2, we can view the GA- Planes family as following a similar strategy with a combi- nation of low-rank and low-resolution approximations. This low-rank plus low-resolution parameterization is generally easier to train, because sparse models must either store large numbers of empty values (high memory) store the loca- tions of nonzero entries and suffer from poorly-conditioned spatial gradients (difficult optimization). Indeed there are volume parameterizations that utilize sparsity, such as point clouds, surface meshes, sparse grids, surfels, and Gaussian splats, but these tend to be more challenging to optimize
Remark. The upper limit of the rank of this matrix ap- proximation is min(n, m, r2), the same as with a convex MLP decoder.
We summarize the maximum attainable ranks of different 2D models in Table 1 (see Appendix A.2.3 and A.2.4 for matrix representations of D(e) + e2) and D(e) @ e2) with convex and nonconvex MLP decoders). Experimental vali- dation of these theoretical results on the task of 2D image compression is provided with a comparison of interpolation schemes in Figure in the appendix.
6

---

Geometric Algebra Planes
(e.g. requiring high memory (Sara Fridovich-Keil and Alex Yu et al., 2022) heuristics and good initialization (Kerbl et al., 2023)).
sampling, strategy for efficiently allocating ray samples. Proposal sampling is the default for K-Planes, but in- clude version without proposal sampling because none of the other models in this experiment use proposal sampling, We also include several ablations of our GA-Planes model: one with only the volume features (similar to a multires- olution version of DVGO (Sun et al., 2022) or Plenoxels (Sara Fridovich-Ken and Alex Yu al., 2022)), one with only the line features (similar to a multiresolution TensoRF. CP), and one with only the line-plane products (similar to a multiresolution TensoRF-VM (Chen et al., 2022)). At large model sizes (~100 million parameters) most models includ- ing GA-Planes perform well. As model size shrinks, only GA-Planes and its line-only ablation reach comparable met- ries as the larger models. Detailed descriptions of parameter allocations for each model are provided in Appendix A.6.
We illustrate this low-rank plus low-resolution interpreta- tion in Figure 2 with a simple experiment, in which we approximate a grayscale image (the astronaut image from SciPy) using either a sum of low-rank and low-resolution components (similar the structure of GA-Planes) or the classic sum of low-rank and sparse components. In this experiment we compute the optimal low-rank components of each model type using the SVD, which corresponds to multiplication of features.
powR
5.2. 3D Segmentation
Low Low
Our experiments for 3D segmentation use opacity masks from the NeRF-Blender lego scene (Mildenhall et al., 2020). We "lift" these 2D segmentation masks to 3D. We compare the linear and nonlinear feature combination versions of GA-Planes and Tri-Planes (Chan et al., 2022; Fridovich- Keil et al., 2023) (models with plane features only).
Sparse
02 Compression Rate
(a) Pareto Frontiers for Image Compression
(b) Low Rank + Low Res
(c) Low Rank + Sparse PSNR 26.26
PSNR 29.60
Figure 2: For a natural image, approximation as sum of low rank and low resolution components (green curve and subfigure b) achieves higher fidelity compared the clas- sic matrix decomposition as sum of low rank and sparse components (blue curve and subfigure c), with the same parameter budget 11%.55% of the original image size, for subfigures and c). The GA-Planes model family general- izes the idea of low rank plus low resolution approximation to three dimensions.
2D Supervision. In this experiment we use 2D tomo- graphic supervision, in which we minimize the mean squared error between the ground truth 2D object segmenta- tion masks and the average ray density at each viewpoint. In Table 2 we report the intersection over union (IOU) metric on thresholded projections from our trained model on test views. These results validate that the GA-Planes architec- ture in Equation (2) works well regardless of convex, semi- convex, or nonconvex formulation-whertt model requires nonconvex decoder for good performance. Note that Tri-Planes with feature concatenation performs on-par with planes with addition, thus, we keep the original formulation with addition for our comparisons.
5. Experiments
the Tri-Planes
5.1. Radiance Field Modeling
Our experiments for the radiance field reconstruction task are built with NeRF-Studio (Tancik et al., 2023) and use all scenes from NeRF-Blender (Mildenhall et al., 2020). We train each volume representation based on the standard pho- tometric loss (Max, 1995; Kajiya, 1986; Mildenhall et al., 2020). Our results are summarized in Figure 3, which re- ports PSNR, SSIM (Wang et al., 2004), and LPIPS (Zhang et al., 2018) for each model as a function of its size. We provide renderings and per-scene pareto-optimal curves in Figure 4 and A.3. We also compare memory and training time requirements in A.7.
3D Supervision. Our second set of 3D segmentation ex- periments leverages direct 3D supervision via Space Carv- ing (Kutulakos Seitz, 1999), using the principle that any ray passing through given 3D coordinate is transparent, the density at that 3D coordinate must be zero. Our results with 3D supervision, in Table 2, parallel those with 2D su- pervision: GA-Planes performs well regardless of convex, semiconvex, or nonconvex formulation, whereas the Tri- Planes model performs decently with a nonconvex decoder but much worse with convex or semiconvex formulation.
Because the photometric loss function is inherently non- convex, we use most expressive GA-Planes parame- terization as defined in Equation (3). We compare this with several popular models as implemented in NeRFS- tudio: Mip-NeRF (Barron et al., 2021), TensoRF (Chen et al., 2022), and K-Planes (Fridovich- Keil et al., 2023). For K-Planes, we include versions with and without proposal
5.3. Video Segmentation
Our video segmentation task is similar to volume segmen- tation with 3D supervision: here the dimensions are x, y, t rather than x, y, 2, and the supervision is performed directly
7

---

Geometric Algebra Planes
612
Mp Hore
331
Khhmn Prgaoaa
PSNM
Gk-fianss Abltton
Dny)
26
330
(brume Dily
OF
Parameters) 3F
Porameters)
Model Parameters)
Figure 3: Results radiance field reconstruction. Nonconvex GA-Planes (with feature multiplication) offers the most efficient representation: when the model is large it performs comparably to the state of the art models, but when model size is reduced it retains higher performance than other models. Here all models are trained for the same number of epochs on all 8 scenes from the Blender dataset, and the average results are shown.
Task 1 (2D Tomographic)
Task 2 (3D Space Carving)
Task 3 (Video Fitting)
Convex Semiconvex
Nonconvex
Convex
Semiconvex
Nonconvex
Convex
Semicoavex
Nonconvex
G.-Pllnes (with
GA-Planes (with 9)
0.875
0.883
0,880
0.932
0.957
0.964
0.913
0.975
0.981
Tri-Planes (planes with o, like K-Planes)
0.877
0.881
0.727
(planes
like (Chan al., 2022))
0.681
0.642
0.557
0.732
Table 2: Tasks 1&2: Intersection over union (IOU) for recovering novel view object segmentation masks across two different experiments: Experiment 1 involves segmentation mask training with 2D tomographic supervision, and Experiment 2 involves segmentation mask training with 3D Space Carving supervision. Task 3: Intersection over union (IOU) for temporal superresolution of segmentation masks in a video, computed on held-out test frames. Models that involve multiplication of features can only be trained by nonconvex optimization.
is slightly reduced under fully convex training, perhaps be- cause the convex model size is slightly reduced due to fusing the decoder parameters into the feature grids. In contrast, the simpler Tri-Plane models perform poorly on this task regardless of training strategy: they fail to learn the temporal sequence of the video, producing masks that focus only on the skateboarder's less-mobile core.
Tni-Planes GA-Planes
Figure 4: Rendering comparison for the mic scene: Ten- soRF on the left, K-Planes the middle, GA-Planes on the right (0.32, 0.39, 0.25 M parameters, respectively). Visual
results other scenes in Appendix A.6.
in 3D using segmentation masks for subset of the video frames (every third frame is held out for testing). Our dataset preparation pipeline uses the skateboarding video and pre- processing steps described Labelbox.co com, which involves first extracting bounding box with YOLOv8 (Jocher al., 2023) and then segmenting the skateboarder with SAM (Kirillov et al., 2023). This is essentially temporal super- resolution task on segmentation masks.
Comex
Semicomex
Noncomve (3
Noncomvex (o)
Ground Truth
Figure 5: Intersection over union (IOU) for predicting seg- mentation masks for unseen frames within video of segmented skateboarder.
5.4. Benefits of Convexity
In most of our experiments, all models (convex, semiconvex, and nonconvex) are large enough that they are able to opti- mize well. However, we highlight a benefit of our convex and semiconvex models that they enjoy more stable opti-
Our results are summarized Figure 5 and Table 2. We find that GA-Planes performs well across convex, semicon- vex, and nonconvex formulations, though its performance
8

---

Geometric Algebra Planes
mization even with very small model sizes. In Figure 6 compare test intersection-over-une
(loU) curves for very small models for our video fitting task (hidden dimension 4 in the decoder MLP, and feature dimensions [d1, d2, ds] =
of
[4, 2] and resolutions r2, "2, ra] = [32, 32, 16] for line, plane, and volume features, respectively). We repeat opti- mization with 10 different random seeds used initialize the optimizable parameters (gating weights for the convex and semiconvex models are fixed) and report the mean and standard deviation for each model's final IoU in Table 3. While the convex and semiconvex models enjoy stable train- ing curves across random seeds, we find that the nonconvex model experiences much more volatile training behavior (completely failing to optimize with some of the random seeds).
0.3
0.1
0.2
30.0 27.3
Figure 6: Test performance throughout training on our video fitting task, for a very small GA-Planes model across 10 random seeds. We find that the favorable optimization landscape of our convex and semiconvex models enables reliable training across seeds, whereas the nonconvex model fails fit any test frame with some of the seeds.
On average, the semiconvex model performs best, followed by the convex model, with the nonconvex model perform- ing worst average. We can also see from the standard deviation values that the convex model is extremely stable with respect to random initialization, as is the semiconvex model. contrast, the nonconvex model is highly unstable, with some seeds failing converge at all.
Mean
Standard Deviation
promising avenues for extending GA-Planes. In our experi- ments, the same first-order optimization algorithm for all models. However, our convex GA-Planes formu- lation is compatible with any convex solver (e.g. cvxpy), and we expect its performance may improve by leveraging these efficient convex optimization algorithms. We demon- strate preliminary benefits of convexity and semiconvexity in terms of training stability in Section 5.4.
Convex GA-Planes 0.639
0.000
Semiconvex GA-Planes
0.716
0.022
Nonconvex GA-Planes 0.567
0.229
Table 3: Mean and standard deviation of intersection over union (IOU) scores for video segmentation over 10 seeds, with small GA-Planes model. Figure 6 visualizes these results over the course of training.
6. Discussion
Impact Statement
In this work we introduce GA-Planes, a family of volume parameterizations that generalizes many existing represen- tations (see Appendix A.1). We specifically focus on two members the GA-Planes family (with concatenation ver- sus multiplication of features), and offer both theoretical interpretation and empirical evaluation of these models. Our nonconvex GA-Planes model shows pareto-optimal perfor- mance in terms of model size and quality on fitting 3D radiance field, while our convex and semiconvex GA-Planes formulations are effective on several 3D linear inverse prob- lems. 2D, we show connections between GA-PL GA-Planes and low-rank plus low-resolution matrix completion model, and derive how this model's rank capacity is affected by various design decisions.
This paper presents work whose goal is to advance the field of Machine Learning. We specifically focus on efficiency of representations and optimization. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
A Acknowledgements
We are grateful to Axel Levy for helpful discussions on space carving, and to Abhishek Shetty for helpful discus- sions on matrix completion. This work was supported in part by the NSF Mathematical Sciences Postdoctoral Re- search Fellowship under award number 2303 178, in part by the National Science Foundation (NSF) under Grant in part by the NSF CAREER Award under Grant CCF-2236829, in part by the U.S. Army Research Of- fice Early Career Award under Grant 1NN-22111--0022 and in part by the Office of Naval Research under Grant 1-0014-24-1-2164.
Limitations. Here we focus on 3D (or smaller) represen- tations, rather than higher dimensions (e.g. dynamic vol- umes), and we demonstrate GA-Planes for reconstruction rather than generation tasks, Both of these extensions are

---

Geometric Algebra Planes
References
Kajiya, T. The rendering equation. In Proceedings of the 13th annual conference Computer graphics and interactive techniques, pp. 143-150, 1986.
Barron, J. T., Mildenhall, B., Tancik, M., Hedman, P., Martin- Bartll,, R., and Srinivasan, P. P. Mip-nerf: A multiscale representation for anti-aliasing neural ra- diance fields. In ICCV, pp- -834-5844. IEEE, 2021. doi: 10.1009//CCC49992220020 00580. URL https: //tpl.oogg/o.1109300..000/// hww.:///w../cVV42222.22 ////////////////////////////22//
Kerbl, B., Kopanas, G., Leimkühler, T., and Drettakis, G. 3d gaussian splatting for real-time radiance field render- ing. ACM Transactions on Graphics, 42(4), July 2023. URL WRLLttss:///ppooaaat sd-gaussian-sspatttin
Hungrrahh/
Brandstetter, J., Berg, R. d., Welling, M., and Gupta, K. Clifford neural layers for pde modeling. arXiv preprint TrXiv:2209.0493:3 04934, 2022.
Kirillov, A., Mintun, Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T.,, Whitehead, S., Berg, A. C., Lo. W.-., Dollár, P., and Girshick, R. Segment anything, 2023. URL 2304.02643.
Cen, J., Zhou, Z., Fang, J., Yang, C., Shen, W., Xie, L., Jiang, D., Zhang, X., and Tian, Q. Segment anything in 3d with nerfs. In NeurlPS, 2023.
Kutulakos, K. and Seitz, S. A theory of shape by space carv- ing. In Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 1, pp. 307-314 vol.1, 1999. doi: 10.1109/CCVV119999999 79129...
Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., Mello, S. D.. Gallo, O., Guibas, L.,, Tremblay, J., Khamis, S., Karras, T., and Wetzstein, G. Efficient geometry-aware 3D generative adversarial networks. CVPR, 2022.
Labelbox.com. Using meta's segment anything (sam) model video with labelbox's model-assisted label- ing. htttps://labelbox.cco
Chandrasekaran, V., Sanghavi, S., Parrilo, A., and Willsky, A. S. Sparse and low-rank matrix decompositions. IFAC Proceedings Volumes, 42(10)):49331449 2009.
netta---eegeee----yychaa-------- -using-
labeling/. Accessed: 2024-10-01.
Chen, A., Xu, Z., Geiger, A., Yu, J., and Su, H. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
Liu, J., Huang, X., Huang. T., Chen, L.,, Hou, Y., Tang, S., Liu, Z., Ouyang, W., Zuo, W., Jiang, J., et al. comprehensive survey on 3d content generation. arXiv preprint arXiv:2402.011666 2024.
Croce, V., Caroti, G., De Luca, L., Piemonte, A., and Véron, P. Neural radiance fields (nerf): Review and potential applications to digital cultural heritage. The International Archives of the Photogrammetrry Remote Sensing and Spatial Information Sciences, 48:453 460, 2023.
Lombardi, S., Simon, T., Schwartz, G., Zollhoefer, M., Sheikh. Y., and Saragih, J. Mixture of volumetric primitives for efficient neural rendering. ACM Trans. Graph., 40(4), jul 2021. ISSN 0730-0301. doi: 10.114533450026.455966. URL URL.ttts::///o..oom 10.1145/34506263300
Dorst, L., Fontijne, D., and Mann, S. Geometric algebra for computer science (revised edition): An object-oriented approach to geometry. Morgan Kaufmann, 2009.
Masero, V., LEÓN-ROIAS, M., and Moreno, J. Volume reconstruction for health care: survey of computational methods. Annals of the New York Academy of Sciences, 980(1):198-2)) 211, 2002.
Ergen, T. and Pilanci, M. Path regularization: A convex- ity and sparsity inducing regularization for parallel relu networks. Advances in Neural Information Processing Systems, 36, 2024.
Max, N. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1 (2)999-108, 1995.
Fridovich-Keil S., Meanti, G., Warburg, F. R., Recht, B., and Kanazawa, A. K- K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., and Geiger, A. Occupancy networks: Learning 3d re- construction in function space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
Intwala, A. M. and Magikar, A. A review process of 3d model reconstruction. In 2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT), pp. 2851-2855. IEEE, 2016.
2019.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T. Ramamoorthi, R., and Ng, R. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Jocher, G., Qiu, J., and Chaurasia, A. Ultralytics YOLO, January 2023. URL https::////// /tts:://iii htwbb.ccmm attralytics/ullralll
10

---

Geometric Algebra Planes
Ming, Y., Yang, X., Wang, W., Chen, Z., Feng, Xing, Y., and Zhang, G. Benchmarking neural radiance fields for autonomous robots: An overview. arXiv preprint arXiv:2405.05526, 2024.
Schönberger, J. L.., Zheng, E., Pollefeys, M., and Frahm, J.-M. Pixelwise view selection for unstructured multi- view stereo. In European Conference on Computer Vision (ECCV), 2016.
Müller, T., Evans, A., Schied, C., and Keller, Instant neural graphies primitives with multiresolution hash encoding. ACM Trans. Graph., 10(4):102:1-102:::: July URL https:
Sitzmann, V., Thies, J., Heide, F., NieBner, M., Wetzstein, G., and Zollhöfer, M. Deepvoxels: Learning persistent 3d feature embeddings. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2019a.
2022. doi: 10. 144444452222233333011111 //
Sitzmann, V., Zollhöfer, M., and Wetzstein, G. Scene repre- sentation networks: Continuous 3d-structure-ave -ware neu- ral scene representations. In Advances in Neural Informa- tion Processing Systems, 2019b.
Oppenheim, A. V. Discrete-time signal processing. Pearson Education India, 1999.
Pilanci, M. and Ergen, T. Neural networks are con- vex regularizers: Exact polynomial-time convex op- timization formulations for two-layer networks. In III. H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7695-7705. PMLR, 13-18 Jul 2020. URL https: / pttppp/eeiigm.m...p.eee..eeeeeee bilanci20a.hter
Sitzmann, V., Martel, J. N., Bergman, A. W., Lindell, D. B., and Wetzstein, G. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
Slapak, E.,, Pardo, E., Dopiriak, M., Maksymyuk, T., and Gazda, Neural radiance fields the industrial and robotics domain: applications, research opportunities and use cases. Robotics and Computer-Intese Integrated Manufac- turing, 90:102810, 2024.
Reiser, C., Szeliski, R., Verbin, D., Srinivasan, P. P., Milden- hall, B., Geiger, A., Barron, J. T., and Hedman, P. Merf: Memory-efficieeee radiance fields for real-time view syn- thesis in unbounded scenes. SIGGRAPH, 2023.
Sun, C., Sun, M., and Chen, H. Direct voxel grid optimiza- tion: Super-fast convergence for radiance fields recon- struction. In CVPR, 2022.
Richter, A., Steinmann, T., Rosenthal, J.-C., and Rupitsch, S. J. Advances in real-time 3d reconstruction for medical endoscopy. Journal of Imaging, 10(5): 2024.
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Ken S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., and Ng, R. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
Ruhe, D., Brandstetter, J., and Forré, P. Clifford group equiv- ariant neural networks. Advances in Neural Information Processing Systems, 36:62922-62990 2023a.
Ruhe, D., Gupta, J. K., De Keninck, S., Welling, M., and Brandstetter, J. Geometric clifford algebra networks. In International Conference Machine Learning, pp. PMLR, 2023b.
Tancik, M., Weber, E., Ng. E., Li, R., Yi, B., Kerr, J., Wang, T., Kristoffersen, A., Austin, J., Salahi, K., Ahuja, A., McAllister, D., and Kanazawa, A. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
Sahiner, A., Ergen, T., Ozturkler, B., Pauly, J. M., Mardani, M., and Pilanci, M. Scaling convex neural networks with burer-monteiro factorization. In ICLR, 2024.
Tewari, A., Thies, J., Mildenhall, B., Srinivasan, P., Tretschk, E.,, Yifan, W., Lassner, C., Sitzmann, V., Martin- Brualla, R., Lombardi, S., al. Advances in neural rendering. In Computer Graphics Forum, volume 41, pp. 703-735. Wiley Online Library, 2022.
Sara Fridovich-Keis and Alex Yu, Tancik, M., Chen, Q., Recht, B., and Kanazawa, A. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
Saragadam, V., LeJeune, D., Tan, J., Balakrishnan, G., Veer- araghavan, A., and Baraniuk, R. G. Wire: Wavelet im- plicit neural representations. In Conf. Computer Vision and Pattern Recognition, 2023.
Udupa, J. K. and Herman, G. T. 3D imaging in medicine. CRC press, 1999.
Uy. M. A., Baualla R., Guibas, L., and Li, K. Scade: Nerfs from space carving with ambiguity depth estimates. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Schönberger, J. L.. and Frahm, J.-M. Structure- ciitieeffroo-moiiioo revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

---

Geometric Algebra Planes
Wang, Z.,, Bovik, A. C., Sheikh, H. R., and Simoncelli, E. Image quality assessment: from error visibility to struc- tural similarity. IEEE transactions image processing, 13(4):600-0) 612, 2004.
Wijayathunga, L., Rassau, A., and Chai, D. Challenges and solutions for autonomous ground robot scene understand- ing and navigation in unstructured outdoor environments: A review. Applied Sciences, 13(17):9877, 2023.
Xu, M., Guo. Z., Wang, A., Bai, L., and Ren, review of 3d reconstruction techniques for deformable tissues in robotic surgery. arXiv preprint 24844:2408.04426 2024.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
12

---

Geometric Algebra Planes
A. Appendix
A.I. Context for GA-Planes
Table summarizes some representative volume mod- els popular in computer vision, and how they relate to GA-Planes along the three-way pareto frontier of model size, expressiveness, and optimizability. In Figure ??, provide overview of our GA-Planes models.
Expressiveness
Optimizability
Model Size
Implicit Neural Representations (INRs) or Coordinate Neural Networks (Mildenhall et al., 2020; Sitzmann et al., 2019b; Tancik et al., 2020; Sitzmann et al., 2020; Sara- gadam al., 2023) parameterize the volume implicitly through the weights of a neural network, typically a mul- tilayer perceptron (MLP) with some modification to over- come spectral bias and represent high frequency content. These models tend to provide decent expressiveness with very small model size; their main drawback is slow opti- mization.
Coordinate MLP (NeRF. SRN)
Voxels (Space Carving, Plenoxels, DVGO)
X
Tensor Factorization (TensoRF, K-Planes)
2
Hash Embedding (Instant-NGiP)
2
Point Cloud Splat (3D Gaussian Splatting)
2
Mixture of Primitives (MVP, MERF)
GA-Planes (Nonconvex)
2 222
Voxel grids (Kutulakos & Seitz, 1999; Sara Fridovich-Kan and Alex Yu et al., 2022: Sun al., 2022; Sitzmann et al., 2019a) are perhaps the most traditional parameterization of volume, where each parameter denotes the function value (density, color, latent feature, etc.) at specific grid cell location within the volume. These voxel values can then be combined into a continuous function over the 3D space by some form of interpolation, following the standard Nyquist sampling and reconstruction paradigm of digital signal processing (Oppenheim, 1999). Voxels offer direct control over expressivity (via resolution) and are easily optimized; their main drawback is memory usage because the number of parameters grows cubically with the spatial resolution.
<<>
GA-Planes (Convex)
GA-Planes (Semi-Convex)
Table 4: Context. All volume models face a tradeoff be- tween memory efficiency, expressiveness, and optimizability, The qualitative categorizations here are based on the tradeoff's achieved by representative example methods listed in each category. Model Size denotes memory usage during training; other methods exist to compress trained models, e.g. for ren- dering on mobile hardware. Optimizability denotes both speed and stability of optimization//raiiing For example, Coordi- nate MLPs tend to train slowly, while Splats train quickly but are sensitive to initialization.
Tensor factorizations (Chen et al., 2022; Chan et al., 2022; Fridovich- Keil et al., 2023) parameterize a 3DD volume as combination of lower-dimension objects, namely vectors and matrices (lines and planes). Tensor factorizations tend to balance the three attributes somewhat evenly, offering decent expressiveness and optimizability while using more memory than an INR but less than high resolution voxel grid.
Hash embeddings (Müller et al., 2022; Tancik et al., 2023) are similar to voxels, but replace the explicit voxel grid in 3D with a multiresolution 3D hash function followed by small MLP decoder to disambiguate hash collisions. They can optimize very quickly and with better memory efficiency compared to voxels; quality mixed with good high-resolution details but also some high-frequency noise likely arising from unresolved hash collisions or sensitivity to random initialization.
Point clouds / spluts (Kerbl et al., 2023; Schönberger & Frahm, 2016; Schönberger et al., 2016) represent volume as collection of 3D points or blobs, where the points need not be arranged on regular grid. They are highly expressive and less memory-intensive than voxels (but still more so than some other methods). They can optimize very quickly but often require heuristic or discrete optimization strategies that result in sensitivity to initialization.
Mixture of primitives (Reiser et al., 2023: Lombardi al., 2021) models combine multiple of the above representation strategies balance their strengths and weaknesses. For example, combining low resolution voxels with high resolution tensor factorization is an effective strategy to improve the expressiveness of tensor factorizations without resorting to the cubic memory requirement of a high resolution voxel grid; this strategy underlies both MERF (Reiser et al., 2023) and GA-Planes.
13

---

Geometric Algebra Planes
We emphasize that all of these existing methods (except perhaps voxels) require nonconvex optimization, often for feature decoder MIP,, and thus risk getting stuck in suboptimal local minima depending the randomness of initialization and the trajectory of stochastic gradients. In practice, described above, some of the prior methods exhibit greater optimization stability than others, though none (except voxels in limited settings) come with guarantees of convergence to global optimality. In contrast, both the convex and semiconvex GA-Planes formulations come with guarantees that all local optima are also global (Sahiner et al., 2024).
Relation to Prior Models. models as special cases:
Without any convexity restrictions, the GA-Planes family includes many previously proposed
NeRF (Mildenhall et al., 2020): D
Plenoxels (Sara Fridovich-Keil and Alex Yu et al., 2022), DVGO (Sun et al., 2022): D(e12))
TensoRF (Chen et al., 2022): D((e1 o e23) @ (e2 e13) @ (e3 e12))
Tri-Planes (Chan et al., 2022): D(e12 + e13 + e2a)
K-Planes (Fridovich-Keill al., 2023): D(e12 G13 C23)
MERF (Reiser et al., 2023): D(e + e13 + e23 + e123)
Of these, all except for TensoRF and K-Planes are compatible with convex optimization towards any convex objective. Note that different models may use different decoder architectures for D. including both linear and MLP decoders and additional decoder inputs such as encoded viewing direction and/or positionally-encoded coordinates.
Convex Neural Networks. Given a data matrix X E Raxd and labels y R", a 2-layer nonconvex ReLU MLP approximates y as
m
(XU))+aJ,
(12)
j=1
where m the number of hidden neurons and U and are the first and second linear layer weights, respectively. (Pilanci Ergen, 2020) proposed to instead approximate y
y Σ D.X(u: w::,
(13)
subject (2D: I,, 2 0 and (2D; -,) 0 for all í. The parameters and in Equation (13) replace the first and second layer weights U and from the nonconvex formulation in Equation (12) (optimal values of U and a can be recovered from optimal values of v and w). Here D, represent different possible activation patterns of the hidden neurons as (D = [Diag(1[Xu 2 0)) E Rd), which is the finite set of hyperplane arrangement patterns obtained for all possible u E Rd We can sample different u's to find all distinct activation patterns (D; ) where P is the number of regions in the partitioned input space. Enumerating
such patterns would yield an exact equivalence with the global minimizer of the nonconvex ReLU MLP in Equation (12), but may be complicated intractable due memory limitations. Subsampling P patterns results in convex program with tractable size, whose solution one of the stationary points of the original non-convex problem (Pilanci & Ergen, 2020). We apply this idea to create convex and semiconvex GA-Planes models by convexifying the feature decoder MI. according to this procedure.
A.2. Proof of Theorems
A general note on proofs. In order to represent matrix M E mm with an implicit model, compute (f) for q (k, Ak E [1....,m}, m). VI (I.....1).. Considering line feature grids with resolutions matching m. n; the features will become = (gı)., e2 = (g2), for q = (k,l) otherwise they will be = p(g)); e2 = <(g2); where p(g)), (((2) now have resolutions m, n after interpolation through Here op can be any interpolation scheme with linear weighting of inputs, e.g. nearest neighbor, (bi)linear, (bi)cubic, spline, Gaussian, sinc, etc. For the simplicity of notation, we omit yp in line feature grids, and only apply the plane feature grid g12, which has lower resolution by design. The
14

---

Geometric Algebra Planes
proofs consider 81. 82 R Reixd xdi and g12 Rraxraxd: (equal feature dimensions, different resolutions), resulting in the matrix representation M xr (the case where
r1). Note that if the interpolation is done by method other than nearest neighbor, this may allow (convex nonconvex) MLP decoder to increase the rank beyond r1. We derive expressions for M implied by different GA-Planes variations in the parts that follow. The coordinate- wise optimization objective (in the case a directly supervised mean-square-error loss) corresponds to minimizing the Frobenius norm the ground truth matrix M and its approximation M.
1.2.1. PROOF OF THEOREM
The forward mapping of the model D(ei + e2) is:
T(4) D(er + e2) a (e) e2) a ((81), + (g2)) Σ
ag((g)) (1, + (g2)..),
(14)
where (g1),. (g1); e xd In matrix form.
di + +Σ (g) i=1
M = Σ a.((g));1 (g)))
Σ i=1
(g)))
(15)
Defining U == (idiag(a), E Xri and V = g2diag(a), V E Rn xd,
this can be expressed as
M U1 Lnxxd
(16)
Note that the resulting matrix M Ru has rank most very limited ceeisivvtty-rgggadeee of the resolution ri. This is because the all-ones matrix is rank and product of matrices cannot have higher rank than either of its factors.
Similarly for the multiplicative representation D(ei ez), the mapping is
di
T(q) = D(e1 o eey) = Tei a e2) = a T (())) (B2)) = Σ ag( (B))).
(17)
where (gi)., (g1); E Rd In matrix form,
M Σ
= g didg (a)g.
(18)
i=1
Defining U gıdiag(a), e x and V = g2: E xd, this can be expressed
M = UV,
(19)
which is the optimal rank-d, decomposition.
A.2.2. PROOF OF THEOREM 2
The forward mapping of the model D(ei + e2 + e122) becomes:
j(q)= = D(ei + e2 + e = a (ee + t2 + +e12) = a ((g)) + (gx), + ((812)..)
(20)
d
((1)), + (g2)(.) + anp(g122
(21)
where (gi),, (gt), (g12) E Rd In matrix form,
M o (11).1 1(B))T + d (11())))
(22)
15

---

Geometric Algebra Planes
Noting that the first term is the same in Equation (15) and defining L :== B120, L E RR2xxr,, we reach the expression
M U d + xd V + (T.))
(23)
since Σ (g1(( <( (g12))) (gg2a), following the linearity of the upsampling function (p. Note that in the definition of I. there is tensor-vector product that effectively takes dot product along the last (feature) dimension.
Similarly for the multiplicative representation D(e) + e12), the mapping is
T(4)= D(e) + e12) a (el + e12)
(24)
a (gg)) (2) ((11)) Σ (
arp(g12)
(25)
In matrix notation, have
d
M
Σ oz ((8))
Σ (g)2)
(26)
Following Equation (18) and using the same definition of L, the final expression becomes
UV + (LL)
(27)
A.2.3. PROOF OF THEOREM
For a 2-layer convex MLP with hidden size h, denote the trainable first layer weights as W hg and the gating weights as We Rexd1 (which are fixed random initialization). We will handle three different cases for merging the interpolated features: multiplication (o), addition (+), and concatenation (o).
The forward mapping of the multiplicative model using convex MLP D(e: o e2) at q (k, is
( 1 ((W(g))) o (B2)r)
(ggi)) (ga)) 2 0])
(28)
d
Σ
Σ
W (a
W (g)) (8)1 20
(29)
where denotes elementwise multiplication (Hadamard product). The resulting matrix decomposition then be written
M
W (8)) (8)),
- 1
W 3.3 (g (1) (g2), 2 0
(30)
Now, we define the masking matrix B, reach the expression from the theorem statement:
Σ j=1
W is (B1),
2 ol and the eigenvectors U, (B:)), V, = (82),
M = Σ Wu B,
(31)
When the model uses additive features as in D(c; 22), and D is convex MLP, the prediction is
v(q) 17, ((W(())) + (g2))) o (((g)) (gz)) 2 0))
(32)
Σ Wir((g))a. + (Bo))o)
Σ W Wes((g))a,) (B2)()) 20
(33)
16

---

Geometric Algebra Planes
The resulting matrix decomposition then be written as
d1 W (is(( + (1(())))) 1
di
M
Σ
W (88() (1 + I (B2))) 2 0
(34)
Defining = 1l - Wi, ((81), + 1r (82)) 2 20. U, (BI), V3 = =B22). we reach the final expression:
M M= W.S + I,, V,) B,
(35)
Finally, we show that concatenation of features results in a very similar expression to Equation (35).
When the model uses concatenated features as in D(e) @ e2), and a convex MLP (with trainable weights W E R4x2d, and fixed gates W E 12x221) the prediction a point q
σ(4) 1, (W((g)) (g2))) 1 [ (gi)) 3 (ga)) 2 0))
(36)
Denoting the weights and gates each as a concatenation of 2 matrices, = (W1 @ W2), W W1, W2, 1., W2 hxxs we have the following expression:
W O W2), where
((4) Σ
W WBi)( + (
1
W (g1) W W215(B2)15
(37)
Following similar steps as for the additive case, we express the matrix decomposition as
M Σ Ugi, + 12 V₁J) Bi,
(38)
where B, 1 E W W1s5(E1),L, + 22., (g))) 2 U0 - V, = (g2),
In all these representations, low-rank matrix is multiplied elementwise with binary mask B,, which makes the maximum attainable rank F1. Thus, with convex MLP decoder, rank of M is limited by the resolution of the feature grids.
A.2.4. PROOF OF THEOREM 4
For standard 2-layer ReLU MLP with hidden size h, denote the trainable first and second layer weights as W Raxd,, E Raxx1 We will handle three different cases for merging the interpolated features: multiplication (o), addition (+), and concatenation (O).
The forward mapping of the multiplicative model using standard nonconvex MLP, D(e; o e2)
T(4) T [W(( (g2)))+
(39)
d
a (B)
(B2)1s
(40)
The resulting matrix decomposition can then be written as
Σ Windir
M
Σ
a
Σ a
(41)
i=1
with U, = (g)) V, (g2),
17

---

Geometric Algebra Planes
When the model uses additive features as in D(e e2), the prediction is
5(4) a" [W [Wi())) + (B2)))l+
(42)
d1
Σ a Σ
Wey((Br)as + (B2)1s)
(43)
The resulting matrix decomposition can then be written as
d1
de [WslU,,, +
M = Σe on Σ
We (B)),, 1 + In (B2)))
E ai
Σ
(44)
again with Un (g)) Vj= = (8a);-
Finally, we show that concatenation of features results in a very similar expression.
When the model uses concatenated features as in D(e @ e2) and D standard nonconvex MLP (with trainable weights W E Rxx2d, and a E RA the prediction point 9 is
(44) = T (W o (8a)))).
(45)
Denoting the hidden layer weights as a concatenation of matrices, W (W1 @ W2), where W1, W2 Raxd, we have the following expression:
d1 WiIj(B))s + W21,(Be)1s
T(4)=
(46)
Following similar steps, we express the matrix decomposition as
h
d
M
=
Oli
Win (81); + W2i, I (82)]
(47)
1=1
Σ 04
Σ
Wis Waili,+WW5lll Weas
(48)
t=1
where U, (g)) V, = (g)
By similar argument to Appendix A.2.3, the maximum attainable rank of all three representations derived here is limited by ri.
A.3. Interpolation Comparison
We present 2D image fitting experiments with the astronaut image from SciPy, validating matrix completion analysis summarized in Table 1, We compare 2D GA-Planes models of the form D(ei e2) (solid colorful lines) and D(e e2) (dotted colorful lines) with the optimal low-rank approximation provided by singular value decomposition (solid black line) in Figure? 7. The same experiment repeated for linear interpolation into the vector (line) features (left) versus nearest neighbor interpolation (right, same as theorems). In this experiment we find qualitatively similar results regardless of the type of interpolation, with slightly better performance using linear interpolation; our experiments we use (bi/tri)linear interpolation.
As expected, we find that linear decoder model with multiplication dramatically outperforms its additive counterpart, which does not improve with increasing model size. We also find that 2D GA-Planes models with MLP decoders can match or exceed the compression performance of the optimal low-rank representation found by singular value decomposition (SVD), especially when using nonconvex MLP. This is a testament to the capacity of an MLP decoder to increase representation rank using fewer parameters than a traditional low-rank decomposition, as well as to the resolution compressibility of natural images.
18

---

Geometric Algebra Planes
45
45
40
40
35
35
PSNR
30
PSNR
30
25
25
Linear
Linear
20
20
Nonconvex MLP
Nonconvex MLP
Convex MLP
Convex MLP
15
15
SVD
SVD
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.8
Model Size Compression Rate
Model Size Compression Rate
Figure 7: 2D image fitting experiments matching the setting of our theoretical results, with a GA-Planes version using only vector (line) features and decoder as specified in the legend. Solid lines denote features combined by multiplication; dotted lines use addition. Left: linear interpolation features; Right: nearest neighbor interpolation of features. For the SVD baseline we use low-rank factors whose resolution matches the target image, so no interpolation is needed (this, and use of nonlinear decoder, is why in some cases 2D GA-Planes can outperform SVD).
A.4. Lower Bounds
Based on the matrix completion theorems and their summary in Table 1, we present lower bounds the Frobenius norm errors of each 2D GA-Planes model. We denote the optimal fitting error of the linear and MLP decoder models by (D(eea)((((gg))) and EMLP(D(f(g)) for different feature combinations f(g). For models with linear decoder,
Etsnear (D(e) e2)) 2 (2
(49)
Etinear (D(e: o e2)) 2 (M)
(50)
Eisnear (D(e) + e12)) 2 aa(M =p(L)),
(51)
where L is a downsampled version of the target M, at the same resolution as the feature grid g12-
For models with convex or nonconvex MLP decoders.
EMLe(D(e; e2)) 2 or, (M)
(52)
EMIL o e2)) 2 or, (M) EMee(D(er e(2)) 2 or, (M ((())).
(53)
(54)
We can see from these bounds that the approximation eroo of model can be reduced dramatically by the introduction of a convex or nonconvex MLP decoder, depending on the singular value decay of the target image M or its high-frequency residual.
A.5. Results for all Nerfstudio- Blender Scenes
In this section, we provide per-scene pareto-optimal curves and qualitative rendering comparisons on various scenes from the Blender dataset. We highlight the superior performance of GA-Planes with limited number of parameters by comparing the smallest K-Planes, TensoRF and GA-Planes models.
19

---

Geometric Algebra Planes
KPann whhhut
SSGM
LPPPS
Popesal Sompling
Panns dblutie
Dey!
Dekume Only
XP Size Parameters)
UP Size Parameters)
12'
Stt Size Porameters)
Figure 8: Results radiance field reconstruction. Nonconvex GA-Planes (with feature multiplication) offers the most efficient representation: when the model is large it performs comparably to the state of the art models, but when model size is reduced it retains higher performance than other models. Here all models are trained for the same number of epochs on the lego scene.
032
032
Hpp Nena
Teeskt
PSNR (73
095
KPiens
LPIPS
Popesal Sampling
GA
Banerian only
L04
line Dny)
693
dtlate (ekkme
Model Size (# Parameters)
Model Size (# Parameters)
Model Size (# Parameters)
Figure 9: Results on radiance field reconstruction. Nonconvex GA-Planes (with feature multiplication) offers the most efficient representation: when the model is large it performs comparably to the state of the art models, but when model size is reduced it retains higher performance than other models. Here all models are trained for the same number of epochs on the chair scene.
A.6. Model configurations used for experiments
A. RADIANCE FIELD MODELING
In our NeRFStudio experiments, we find that because GA-Planes contains features with different dimensionalities (line, plane, and volume), experiences little loss in quality over wide range of model sizes. For small model sizes, we allocate most of the model memory to the line features, since their spatial resolution grows lincarly with parameter count. As the parameter budget grows, we allocate more parameters to the plane features, whereas the performance of the line-only model stagnates with increasing size. Similarly, as model size grows even further we allocate more parameters to the volume features, whose memory footprint grows cubically with spatial resolution.
The original K-planes model uses proposal networks with different resolutions (as noted in Table 5) and a fixed channel dimension of for both. The resolutions and channel dimensions for either K- K-planes model (with vs. without proposal sampling) refer 72 and d2, respectively. TensoRF model resolutions and channel dimensions way, since their feature combination dictates that di d2 and they initialize the line and plane grids with the same resolution. The only nuance is that TensoRF constructs separate features for color and density decoding. Hence, the channel dimensions for density and color features are listed. Instead of a multiresolution scheme, TensoRF starts from the base resolution of r1 12 = 128 and upsamples the grids to reach the final resolutions on Table 5. Resolutions listed under GA-Planes should be interpreted as 12, ra]; channel dimensions as [di, d2, dg]. For all models that use the multiresolution scheme, the base resolutions (i.e. [r1. r2, r3) are multiplied with the upsampling factors. For instance, base resolution [r1. T2, r3] with channel dimensions [d1, d2, da] and multiresolution copies [m,,,72, ma] will generate the grids of GA-Planes as follows: Linear feature grids g1, B2. E3 will have the shapes {[1rr1d1] d2), [m271, di], [my71, di]}; plane grids g12, 823, 813 will have the shapes {[m172, m2,2, da), [m222, m222, dz), [mzr'2, m332, da]}; J. and the volume grid g123 will have the shape [r3, T3, da). Although we don't t use multiresolution copies for the volume grid in GA-Planes, we do use multiresolution for the volume-only GA-Planes ablation. The resolution for that model refers to r3. and the channel dimensions are also allowed to vary for each resolution (unlike other variants with multiresolution, where the feature dimension is fixed across resolutions). we denote these varying feature dimensions as (dea, dab. d the multiresolution copies of the volume grids
be interpreted in similar
20

---

Geometric Algebra Planes
1.90
Hg--eet
7SHA
6370
UPMPS
Popesall Sompling
KPann withut
Plans dblutie,
6190
660
Dey!
Dekume Only
1P
12'
Size Parameters)
Size Parameters)
Porameters)
Figure 10: Results radiance field reconstruction. Nonconvex GA-Planes (with feature multiplication) offers the most efficient representation: when the model is large it performs comparably to the state of the art models, but when model size is reduced it retains higher performance than other models. Here all models are trained for the same number of epochs on the mic scene.
032
Kaanee wihout
PSWR
SS394
LAPS
Line
Gk abarce
Ph-ann iblatee
Model Size (# Parameters)
Model Size (e Perameters)
Model Size Perameters)
Figure 11: Results on radiance field reconstruction. Nonconvex GA-Planes (with feature multiplication) offers the most efficient representation: when the model is large performs comparably to the state the art models, but when model size reduced retains higher performance than other models. Here models are trained the same number epochs on the drums scene.
will have the shapes {mmrr,, mirs, m173; dsa], [m2rs, mars, m2r3, daal, [ma's, myrs, mars, d3e]).
A.6.2. 3D SEGMENTATION
GA-Planes model uses feature dimensions [d1, d3] = [36, 24, 8] (with @) or [d1, d2, ds] [25, 25, 8] (with o) and resolutions r2,,
F2, ra] = [188, 32, 24). Multiresolution grids are not used for this task since density prediction can be achieved by a simpler architecture. The model size is 0.22 M. Tri-Planes model has the feature dimension d2 = 4, and resolution "2 128 resulting
total number of parameters of 0.2 M. Note that fix these sizes across (non/semi)convex
formulations, which causes slight variations in the size of the decoder, however, the grids constitute the most number of parameters, making this effect negligible.
A.6.3. VIDEO SEGMENTATION
GA-Planes model uses feature dimensions [dj, d2, d3] = [32, 16, 8) and resolutions 1r,,72,73) 72, = [128, 128, 64]. When the
features are combined by multiplication the nonconvex model, d1 = d2 = 16. Multiresolution grids are not used for this task. The model size is 2.9 Tri-Planes model has the feature dimension d2 = 59, and resolution rg = 128 resulting in a total number of parameters 2.9 M.
A.7. Memory and Compute Times
plot the training times, memory sizes and PSNR metrics GA-Planes and K-Planes for radiance field reconstruction task Figure 20, using the same configurations as in Figure 3. Our method does not incur a memory or compute overhead, however the training times vary due to the choice of feature dimensions. GA-Planes and K- K-Planes achieve the same training time with the same feature dimension, and both models be configured to achieve faster training by reducing feature dimensions.
21

---

Geometric Algebra Planes
Hp--eet
KPann
KPann wthou
PSN
LRPP
Gk GhPann
okPann Abotte
Gk Ablutce
Dekune Onl)
Figure 12: Results on radiance field reconstruction. Nonconvex GA-Planes (with feature multiplication) offers the most efficient representation: when the model is large it performs comparably to the state of the art models, but when model size reduced retains higher performance than other models. Here all models are trained for the same number of epochs on the holdog scene.
351
021
PSWR (T)
KPamm
KPaan wihout
SSIM
GA Gh-Panns
Pk-ane Ablutte
604
GhPann Ablutin
(ekume Ofly)
Size Parameters)
Size Parameters)
Model Size Parameters)
Figure 13: Results on radiance field reconstruction. Nonconvex GA-Planes (with feature multiplication) offers the most efficient representation: when the model is large performs comparably to the state the art models, but when model size reduced retains higher performance than other models. Here all models are trained for the same number of epochs on materials scene.
6:0
Hiphectt
PSHR (T)
Krenn
KPann witout
SSJM
LPIPS
Pano
Gk Panan Ablutie
GA-Pas dblatie
(ekkma Only)
Parameters)
Size Parameters)
Model Size
Figure 14: Results on radiance field reconstruction. Nonconvex GA-Planes (with feature multiplication) offers the most efficient representation: when the model is large it performs comparably to the state of the art models, but when model size reduced retains higher performance than other models. Here all models are trained for the same number epochs on the ficus scene.
FSHR (1)
Kranns
KPaans witout
SSJM
Gk Panns
GA Fann Ablutice
434
832
Panss Ablutic
830
Dekume Orly)
Model Size Parametors)
Model Size Parameters)
Model Size Parameters)
Figure 15: Results on radiance field reconstruction. Nonconvex GA-Planes (with feature multiplication) offers the most efficient representation: when the model is large performs comparably to the state the art models, but when model size reduced retains higher performance than other models. Here all models are trained for the same number of epochs on the ship scene.
22

---

Geometric Algebra Planes
Figure 16: Rendering comparison for the chair scene: TensoRF on the left (0.32 M parameters), K-Planes in the middle (0.39 M parameters), GA-Planes on the right (0.25 M parameters).
Figure 17: Rendering comparison for the mic scene: TensoRF on the left (0.32 parameters), K-Planes in the middle (0.39
M parameters), GA- CA-Planes on the right (0.25 M parameters).
23

---

Geometric Algebra Planes
ae
Figure 18: Rendering comparison for the lego scene: TensoRF on the left (0.32 M parameters), K-Planes in the middle
(0.39 M parameters), GA-Planes on the right (0.25 M parameters).
Figure 19: Rendering comparison for the materials scene: TensoRF the left (0.32 M parameters), K-Planes in the middle (0.39 M parameters), GA-Planes on the right (0.25 M parameters).
24

---

Geometric Algebra Planes
Model
Resolutions
Channel Dimen-
Multiresalution
Proposal Network
Number of model
sions
Resolutions
parameters (M)
4
[1, 2, 4]
[32, 64]
0.390
32
8
[1,2,4]
[32, 64]
0.649
64
4
[1
[64. 128]
1.533
K-plane
64
8
[1,
[128, 256]
4,041
[1,
[128, 256]
6.107
128
[1.244
[128, 256]
10.234
128
16
[1.2.4]
[128, 256]
18.493
32
4
[1,2,4]
0.298
40
4
[1,2.4]
0.444
[1,
1,073
K-plane without proposal sampling
8
[1,2,4]
2. 108
100
6
[1,2,4]
3.822
128
10
[1.2,4
10.367
129
16
1.2.4
16.824
128
[2, 41
0.320
256
[2. 4]
1.207
256
[6. 8]
2.786
256
[12, 12]
4,760
TensoRF
300
[12, 16]
7.609
300
[16. 24]
10.860
300
[32, 32]
17.362
300
[16, 48)
17.364
[200. 4]
[32, 32,
[1.2,
0.254
[200, 4]
[32, 32,
11,
0.351
[200, 16, 8]
[32, 32, 4]
[1,2,4
0.740
GA-plane
[200, 32, 8]
[16, 16, 4]
[1,2, 4]
1.164
[100, 100, 16]
[6, 6,
[1,
3.874
[200. 128, 32]
[10. 10.
[1.2.4
10.681
[200, 128. 32]
[16. 16.
[1,244
16.908
32
4
[1,2.4]
0.301
64
4
[1.2.4]
1.078
GA-plane ablation-VM VM
128
4
[1,2,4]
4, 180
128
[1,
6.251
128
2,44
12.465
200
32
[1,2,4]
0.196
GA-plane ablation-CP
200
64
[1.2.4]
0.355
18
[3, 6]
[1,2,]]
1,236
[4, 4,
[,,
2.778
GA-plane ablation-volume
[4, 4. 6]
[1, 11,2.31
6.529
32
[4. 6.
(1
8.824
Table 5: Model configurations used for the radiance field modeling task on the Blender dataset.
25

---

Geometric Algebra Planes
Model Comparison: Size vs PSNR vs Training Time
23 min
46 min
32.0
29 min
19 min
31.5
min
20 min
16 min
PSNR (dB)
min
30.5
15 min
30.0
29.5
1/7
Models
GA- GA-Planes
15 min
K-Planes
28.5
20
50
Model
We plot the training time, memory and PSNR values of GA- GA-Planes and the strongest baseline, K-Planes. The sizes of the points are proportional training times, as written on the plot. Note that the training durations are directly affected by the selected feature dimensions, and GA-Planes indeed achieves the same speed as K-Planes if the feature dimensions set to be equal). Here show the times required for the model configurations used in radiance field reconstruction task.
Figure 20:
be configured accordingly to enable faster training (and
26


Reviews:
[{"review_id": "QLh4LsQefA", "reviewer": "Reviewer_hYwc", "summary": "This paper provides an analysis of the mixture of the n-dimensional (n<3) voxel representations for learning neural fields. As the authors mentioned, this voxel representation can be line, plane, or volume, which can be viewed as a low-rank or low-resolution representation to encapsulate the target scenes or images. By providing 4 theorems, the authors analyze the expressiveness of each different model design which is consistent with the experimental results. Moreover, in my understanding, this analysis and the results are also consistent with the related studies, such as K-Planes, TensorF, etc.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review.\n\nResponse to 1 (comparison with volume-based method): The pink dashed line marked as “GA-Planes ablation (volume only)” is a volume-based method. In the notation we use, this is D($e_{123}$). We believe the inferior performance is caused by the coarser resolution of the 3D grid compared to the higher resolution that 2D and 1D grids can afford with the same total parameter count. Increasing the resolution of the volume features is expensive in terms of memory. Based on our 2D theory, the use of an MLP decoder enables high effective rank (limited by the resolution) representations even through “low-rank” components such as lines. Thus, the volume-only model does not achieve the expressivity of a combination of high-resolution lines/planes at the same memory budget.\n\nResponse to 2 (local minima vs model expressiveness for volume-based vs factorized and implicit models): We note that for NeRF training, the objective function itself is nonconvex, so regardless of the volume parameterization there is a risk of getting stuck in a local minimum. However, if we consider convex objectives such as shape/video/image fitting, then a 3D grid or a concatenation of tensor features (e.g. our convex GA-Planes model) should be able to optimize globally, whereas a tensor factorization involving feature products (e.g. our nonconvex GA-Planes model, and K-Planes) would still be nonconvex. As described above, we believe that the primary reason behind the performance improvement we observe compared to a 3D grid is based on the difference in resolution, which improves the representation capacity or effective rank of our models, rather than a difference in optimizability.\n\nResponse to 3 (analysis of MLP-based models): With a purely MLP-based model, we cannot define an equivalent matrix factorization optimization problem like the ones we provide in our theoretical analysis of GA-Planes. However, some prior works [1] have analyzed the NeRF model with Fourier features through the lens of neural tangent kernels (NTKs).  \n[1] https://arxiv.org/pdf/2006.10739", "claims_and_evidence": "Yes. The claims in the paper are precise and correct. The authors provide four theorems that progressively investigate the expressiveness of the models in terms of the matrix rank. The first theorem is the most basic model that simply consists of line grids with a linear decoder, which is close to the matrix decomposition as the authors mentioned. The last theorem is the combination of the low-rank and low-resolution grids with non-linear MLP layers that are understood as the most expressive representations compared to the previous models explained in previous theorems. I fully understand these theorems and I think it is correct and well analyzed.", "methods_and_evaluation_criteria": "Based on these four theorems, the authors provide a GA model that represents n-dimensional data (2D images or 3D scenes) with low-rank and low-resolution grid representation with non-convex MLP decoders. Figure 2 and Figure 3 support the efficacy of this method and the evaluation metrics are properly chosen.\n\nMoreover, the authors conduct 2D/3D segmentation experiments in Section 5.2, and video segmentation in Section 5.3. By doing so, the authors provide supportive results in 2D/3D/2D+temporal domains. While there are no experiments handling n-dimensional data, it is fine with me.", "theoretical_claims": "Overall, the theorems are correct. In the first theorem, the authors present the basic model using two matrices U, V which is the same as matrix decomposition. The second theorem is the replacement of matrix multiplication with matrix addition. This is to provide an analysis that the matrix addition has the limitation of expressiveness due to the upper bounded matrix rank. The third theorem is an extension of the first theorem using two-layered MLP decoder. The last theorem is the generalized version of the third theorem, which utilizes the standard MLP layers. \n\nFor each step, the authors provide theoretical analysis in terms of the model expressiveness by checking the matrix rank, and these results are summarized in Table 1 of the manuscript.", "experimental_designs_or_analyses": "The authors conduct (1) image reconstruction in Figure 2, (2) radiance field modeling in Figure 3, and 2D/3D/Video segmentation in Table 2. By doing so, the authors well prove their claim about the model expressiveness in different domains, which looks sound and reasonable to me. Overall, the authors provide qualitative and quantitative results, and the results well support the author's claim.\n\nEspecially. the results in Figure3 well explain the necessity of the low-rank and low-resolution grid representation. The authors compare the proposed GA model with two popular low-rank grid-based methods, TensorF, and K-planes, and the results from the proposed method outperform the previous methods. This is a strong clue to support the authors' claims.", "relation_to_broader_scientific_literature": "So far, the recent studies in the neural fields using low-rank and low-resolution grids only provide empirical results without theoretical analysis in their model designs. At this point, this paper provide profound and theoretical reasons that align with the recent studies. I believe that this paper well describe their understanding in the hybrid grid representation and this will be beneficial to the researchers in this domain.", "essential_references_not_discussed": "Overall, this paper well addresses the generic issue in the hybrid grid representation. I cannot find anything that the authors missed. In the supplementary material, the authors provide additional results and technical comparison with various methods, such as the hash-grid-based method [InstantNGP]. So, I believe there is nothing to discuss more.", "other_strengths_and_weaknesses": "This paper is well written. Providing four theorems is highly admirable and I really enjoyed reading this manuscript. It well supports the results in this paper as well as the results in many recent studies, which claim why hybrid grid representation can be beneficial to their downstream tasks. \n\nI have a few concerns about this work.\n\n(1) Can the authors provide a comparison with volume volume-based method as well in Figure 3? Theoretically speaking, the full-volume grid representation can be the most expressive compared to other grid-based methods including the proposed GA model. However, it is widely known that the volume-only methods tend to fall into the local minima with relatively low fidelity results compared to the MLP-only methods, or MLP-voxel hybrid methods [MipNeRF, ZipNeRF]. For example in the paper of [MonoSDF]. The MLP-only methods outperform the volume grid-based methods in the surface reconstruction task.\n\n(2) In this perspective, I believe the low-rank problem can become a rescue to avoid falling into such a problem. While expressiveness is an important factor, it is not the only one that affects the final rendering quality in my opinion. __I hope that the authors provide an opinion about this issue in the rebuttal.__\n\n(3) I wonder whether we can provide the expressiveness of the MLP-only method, such as Mip-NeRF. Can the authors provide their own analysis of this method?", "other_comments_or_suggestions": "I described my questions in the above section. Please refer to the one.", "questions_for_authors": "I described my questions in the above section. Please refer to the one.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "There is no ethical issue in this paper."}}, {"review_id": "d39K3hukwf", "reviewer": "Reviewer_BtXY", "summary": "The paper reviews existing literature on INRs, noticing that each method presents a trade off between its representation power and its size and optimizability. Based on Clifford algebra, Geometric Algebra Planes are introduced, generalizing some of the existing approaches which use 2 or 3-dimensional feature grids. GA uses a combination of grids at different dimensions and resolutions, and using different combination methods (e.g. multiplication and/or concatenation), resulting in convex, semi-convex and non-convex models. A theoretical derivation describes the behaviour and bounds of the proposed approach, also providing practically useful insights. Experiments with neural radiance fields and segmentation are shown to validate the method.\n\n#Update after rebuttal\nThe rebuttal addressed most of my concerns. I am still not convinced of the validity of the video segmentation experiment, but overall I lean towards acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and thoughtful engagement with our work. We address your concerns below.\n\nResponse to E1 (other baselines and nonconvex decoders): Thank you for referencing [1]. In our 3D segmentation experiments, we use a different dataset compared to data types mentioned by the survey paper. The task we consider is to lift the 2D segmented images of a particular scene to get the 3D segmentation of the object, which is very similar to the normal radiance field reconstruction task, with the only difference being the lack of color prediction. We chose this task as it has a linear forward operation and can be formulated as a convex optimization problem with the use of a convex loss function (like mean-squared error). In the referenced paper, the segmentation is done on different representations of the 3D space. By focusing on optimizing a method for reconstructing 3D scenes from 2D images, we allow for the use of any highly performant, open source pre-trained segmentation model to be used to produce binary masks to be lifted to 3D by our method. Thus, any improvement of 2D segmentation models can be directly reflected in our 3D segmentation output. Regarding decoders, our GA-Planes representation can be used as a feature embedding with any desired decoder including existing INRs as decoders; indeed our nonconvex GA-Planes model uses a standard nonconvex MLP decoder. For generative modeling, a similar approach like EG3D paper (using Tri-Plane representation) could be used with GA-Planes as the underlying representation (this is an exciting direction for future work).\n\nResponse to E2 (clarification on video segmentation): The video segmentation task we consider is the same as 3D segmentation with 2D+time (xyt) replacing 3D (xyz). We show performance on temporal superresolution in this task, which is interpolating missing frames, similar to rendering missing view angles. Again, here we are taking per-frame segmentation masks as training input and our goal is to essentially interpolate these 2D masks into a coherent 3D mask across the video (including missing test frames).\n\nResponse to E3 (shape/image representation): Our 3D segmentation experiment is performing shape fitting (with either direct 3D supervision via space carving, or indirect 2D “tomographic” supervision); we will clarify our description of this task in the revision. We also include image fitting experiments to support our theoretical analysis (figure 7 in appendix of original submission).\n\nResponse to GA references: Thank you for pointing us to these models using geometric algebra. We are happy to include them in our revised related works discussion.\n\nResponse to weaknesses:\n\nW1 (clarification on existing experiments): Please refer to our responses above to E1, E2, and E3. \n\nW2 (speed and memory): No, our method does not come with a speed or memory overhead. We compare the number of parameters (memory) and performance metrics such as PSNR in figure 3. We also added a plot comparing memory and training times of GA-Planes and K-Planes with the configurations as in figure 3 (https://imgur.com/a/Kgm12Pk). Note that the variations in training time are directly influenced by the input feature dimension that gets decoded by the MLP. Comparing GA-Planes and K-Planes with the same feature dimensions, their training times are on par. As noticeable on the plot, GA-Planes had larger feature dimensions on some models, leading to longer training times (we selected the configurations to optimize memory vs. PSNR, which we list on appendix table 5).\n\nResponse to suggestions: Thank you for noticing the typo about referring to figure 1. We will put the full figure in the main text.", "claims_and_evidence": "The claims are mostly supported by evidence. Experimental evidence could be improved (see below).", "methods_and_evaluation_criteria": "The proposed method appears appropriate for the described problem.", "theoretical_claims": "I am familiar with the described problem, but not an expert in the theory behind it, therefore I am unable to verify the proofs for the theoretical claims.", "experimental_designs_or_analyses": "The experiment on NeRF is valid, and shows a comparison with the most important baselines, showing good performance across multiple model sizes. \nE1) The 3D segmentation experiment also seem valid and show that GA-planes supersedes Tri-planes. However, to fully convince the reader, other baselines should be included as well. Examples can be found in [1]. While achieving state of the art might not be the goal of the work, showing that the GA-planes can improve existing works seems to be beneficial. Can GA-planes be used in conjunction with existing models used as (probably non-convex) decoders?\nE2) The video segmentation task is unclear: it shows that GA-planes is better than tri-planes, however I do not understand the significance of a segmentation task where the ground truth comes from an existing segmentation model. \n\nE3) Experiments on shape/image representation could also show the benefit of the method, and would have been an important addition to the paper to show the general validity of the method. \n\n\n[1] Deep Learning Based 3D Segmentation: A Survey, Yong He, Hongshan Yu, Xiaoyan Liu, Zhengeng Yang, Wei Sun, Saeed Anwar, Ajmal Mian", "relation_to_broader_scientific_literature": "The contributions seem sufficiently contextualised, both in their theoretical framework and among recent baselines, except on segmentation (see concerns above).", "essential_references_not_discussed": "Potential additional works needed for the segmentation experiments (see above).\nA related works section on Clifford algebra-related methods should also be included [2] [3] [4], given the claim of being \"the first use of GA in neural volume models\" \n\n[2] Clifford Neural Layers for PDE Modeling, Johannes Brandstetter, Rianne van den Berg, Max Welling, Jayesh K. Gupta\n[3] Clifford Group Equivariant Neural Networks, Ruhe et al.\n[4] Geometric Clifford Algebra Networks, Ruhe et al.", "other_strengths_and_weaknesses": "Strengths:\nS1) The paper is well written\nS2) The analyses are well made and offer good insights with also practical applications, as in section 4.2. Figure 3 is also very informative, including multiple methods at different sizes.\nS3) The method seems well grounded and generalises common approaches such as Tri-planes\n\nWeaknesses:\nW1) The experimental weaknesses raised above concern me about the practical usability of the method, so I would like the authors to address those concerns.\nW2) Does the method come with a speed and/or memory overhead? Given the claims of performance/memory optimality, this should be addressed.", "other_comments_or_suggestions": "Figure 1 is referenced in the text as containing 3 rows but it only has one. The text is referring to the supplementary, I imagine. Having a full (but smaller) figure in the main text would be beneficial.", "questions_for_authors": "I would like the authors to address my concerns about experimental design and memory/speed overhead.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LdHYTryDAH", "reviewer": "Reviewer_YmR7", "summary": "The paper aims to improve grid-based representations for neural fields.\nInspired by principles from geometric algebra, the paper introduces a set of formulations, specifically, convex and semi-convex representations, to enhance the expressiveness and efficiency of grid-based neural fields.\nExperimental results on several tasks, including 3D novel view synthesis, segmentation, and video segmentation, demonstrate that the proposed method can improve the efficiency of the representations.\n\n## update after rebuttal\nNo major concerns were raised, and most of the concerns were resolved during rebuttal. Thus, I would like to keep my initial score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful engagement with our work.\n\nResponse to other comments and suggestions: Thank you for noticing the inconsistency with figure 1. This is a typo, the full figure is in the appendix (figure 6) due to space limitations in the main text.\n\nResponse to questions:\n\nQ1 (full architecture figure in appendix): We did not have enough space.\n\nQ2 (performance of Tri-Planes with concatenation): The performance is very similar to original Tri-Planes (with feature addition). For instance, in the 3D segmentation with 2D supervision task (task 1), Tri-Planes with concatenation using the same channel dimensions and resolutions (leading to slightly more parameters as the input feature size to the MLP decoder is increased) achieves the following IOU results: 0.691 (convex), 0.870 (semiconvex), 0.868 (nonconvex). The results achieved by addition of the features were: 0.681 (convex), 0.868 (semiconvex), 0.863 (nonconvex). Since the difference is very minor, we present Tri-Planes in its original form.\n\nQ3 (guidelines for when to use convex, semi-convex, vs. nonconvex GA-Planes): The radiance field reconstruction task is inherently nonconvex due to the nonlinear forward model. Thus, using a semiconvex or convex model does not provide any theoretical or practical advantages. For the other 2 tasks, convexity and semi-convexity improve stability with respect to random initialization and training procedure. However, empirically we find that nonconvex models also work well once model size is large enough; convexity is most essential for small models that are most sensitive to random initialization. Convex models can also be trained very quickly with specialized convex solvers (a future direction for our project). Semiconvex models don’t have fast, dedicated solvers like convex models, but they also achieve the global minimum of the objective and are thus more stable compared to nonconvex models with similar performance.", "claims_and_evidence": "The paper claims that incorporating geometric algebra concepts into grid-based neural field representations can enhance expressibility.\nTo support this, the authors provide experimental results on multiple tasks.", "methods_and_evaluation_criteria": "Although the idea of combining lines, planes, and cubes for 3D representations is not new in this domain, the paper introduces intriguing idea of connecting them to Geometry algebra and convexity.\nHowever, the transition to the introduction of convexity seems abrupt.\nIn addition, it is not entirely clear whether the choice among these formulations is task-dependent or if there is a general guideline for their application.", "theoretical_claims": "Theoretical claims seem valid.\nThe analysis, based on the 2D setting, is extended to 3D tasks through experiments.", "experimental_designs_or_analyses": "The experimental designs appear to be standard.", "relation_to_broader_scientific_literature": "This work is closely related to prior research on efficient grid-based representations, such as TensoRF, K-Planes, and MERF.\nThe proposed method could provide interesting direction to convexity and other untouched aspects in this domain.", "essential_references_not_discussed": "The paper cites all essential references.", "other_strengths_and_weaknesses": "No comments", "other_comments_or_suggestions": "In Section 3.1, the paper states, \"The specific models we use for nonconvex, semiconvex, and convex optimization are illustrated in Figure 1\", yet only the nonconvex model is shown.", "questions_for_authors": "Q1: Is there a specific reason for including the overviews of the three architectures in the supplementary rather than in the main paper?\nQ2: In Table 2, how is the performance of Tri-Planes model with concatenation?\nQ3: In 2D representation, novel view synthesis, and novel view object segmentation, non-convex representations seem to perform best, while semi-convex seem to perform best for video segmentation tasks. Could you provide more detailed guidelines or discussion on how one should select among nonconvex, semi-convex, and convex representations for a given task?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission got 3 positive recommendations eventually. The reviewers were initially concerned about the evaluations, technical details, and the analysis. The authors did a good job in their rebuttal and addressed most of these concerns. The reviewers did not actively engage in a discussion although the AC tried to initiate one. The AC read through the submission, all reviews, and rebuttals. The AC agreed with the reviewers and values the idea. Per this, the AC made a decision of acceptance. The decision was also approved by the senior AC.
Decision: Accept (poster)