CoordX: Accelerating Implicit Neural Representation with a Split MLP Architecture
Authors: Ruofan Liang
Hongyi Sun
Nandita Vijaykumar
Venue: ICLR 2022 Poster (2022)

Abstract: Implicit neural representations with multi-layer perceptrons (MLPs) have recently gained prominence for a wide variety of tasks such as novel view synthesis and 3D object representation and rendering. However, a significant challenge with these representations is that both training and inference with an MLP over a large number of input coordinates to learn and represent an image, video, or 3D object, require large amounts of computation and incur long processing times. In this work, we aim to accelerate inference and training of coordinate-based MLPs for implicit neural representations by proposing a new split MLP architecture, CoordX. With CoordX, the initial layers are split to learn each dimension of the input coordinates separately. The intermediate features are then fused by the last layers to generate the learned signal at the corresponding coordinate point. This significantly reduces the amount of computation required and leads to large speedups in training and inference, while achieving similar accuracy as the baseline MLP. This approach thus aims at first learning functions that are a decomposition of the original signal and then fusing them to generate the learned signal. Our proposed architecture can be generally used for many implicit neural representation tasks with no additional memory overheads. We demonstrate a speedup of up to 2.92x compared to the baseline model for image, video, and 3D shape representation and rendering tasks.

Published as a conference paper at ICLR 2022
COORDX: ACCELERATING IMPLICIT NEURAL REPRE- SENTATION WITH A SPLIT MLP ARCHITECTURE
Ruofan Li
Hongyi Sun', Nandita Vijaykumar'2
"Department of Computer Science, University of Toronto
Vector Institute, Canada
fruofan, nandita ecs. twronto.to.......... edu, sangyi.suemaal..uue ott.oeo
ABSTRACT
Implicit neural representations with multi-layer perceptrons (MLPs) have recently gained prominence for wide variety of tasks such novel view synthesis and 3D object representation and rendering. However, a significant challenge with these representations is that both training and inference with an MLP over a large number of input coordinates to learn and represent an image, video, or 3D object, require large amounts of computation and incur long processing times. this work, we aim to accelerate inference and training of coordinate- pardinate-based MLPs for implicit neural representations by proposing a new split MLP architecture, Co- ordX. With CoordX, the initial layers are splil to learn each dimension of the input coordinates separately. The intermediate features are then fused by the last layers generate the learned signal at the corresponding coordinate point. This signif- icantly reduces the amount of computation required and leads to large speedups in training and inference, while achieving similar accuracy as the baseline MLP. This approach thus aims at first learning functions that are a decomposition of the original signal and then fusing them generate the learned signal. Our proposed architecture can be generally used for many implicit neural representation tasks with no additional memory overheads. We demonstrate speedup of up 2.92x compared to the baseline model for image, video, and 3D shape representation and rendering tasks.
INTRODUCTION
Traditional discrete representations, including volumes or point clouds, have long been used for tasks such 3D object representation and novel view synthesis. Until recently, implicit neural representations have emerged popular alternative approach to those tasks. These implicit neural representations leverage multilayer perceptrons (MLPs) to represent images, videos, 3D objects, etc (Mildenhall al., 2020; Sitzmann et al., 2020: Tancik et al., 2020). These MLPs (coordinate- based MLPs, coordMLP for short) take low dimensional coordinates (c.g., coordinate position or time index) as input and predict the value of learned signal (e.g.. color, depth, shape, etc.) of each coordinate point. The coordinate-based MI.P learns an implicit, continuous, and differentiable function that maps input coordinates various signals.
There are several benefits to using an implicit neural representation instead of a traditional discrete representation. First, is potentially more memory efficient. Unlike the discrete representation where the granularity of the represented object is limited by the grid resolution, the granularity of an implicit neural representation can be freely adjusted by choosing different resolutions of input coordinates while not exceeding the representation capacity of the learned MLP. Second, the fully differentiable learning process allows the MI. to learn complex signals from sparsely available data points to reconstruct high-quality images or objects. Recent research has demonstrated the effective- ness of coordMLPs for a wide range signal fitting or learning tasks such as image super-resolution (Chen et al., 2021), 3D shape representation (Park et al., 2019; Mescheder et al., 2019; Gropp al., 2020), novel view synthesis (Mildenhall et al., 2020; Martin-Brualla et al., 2021: Schwarz et al., 2020) and photo-realistic 3D scene editing (Niemeyer & Geiger, 2021). The search for more accu- rate and generalizable implicit neural network architectures and methodologies is an active area of research.

---

Published as a conference paper at ICLR 2022
However, key challenge of implicit neural representation is the significant computation demand for both inference and training. For inference, the MLP has to be queried with each positional point to generate the learned signal at the corresponding position. Using one 1080 GPU, rendering 1024 x 1024 sized image on 5-layer coordMLP takes 0.2 seconds, and rendering an image of size 400 x 400 from novel view with the NeRF model takes 18 seconds. Both tasks perform millions of MLP queries. Similarly, training an MLP for neural representation is very conputation-intensit task as each MLP requires full training passes over a large number of sample points from the original signal. On high-end RTX GPU, it takes more than 10 minutes to learn 512 x 512 image using a 5-layer coordMLP and more than 4 hours to reconstruct a low-resolution scene with NeRF.
Several recent works aim to improve the speed of inference and/or training for coordinate- based MLPs for novel view synthesis tasks (Liu et al., 2020; Neff et al., 2021; Garbin et al., 2021; Yu et al., 2021) or object representation (Takikawa et al., 2021). These works leverage spatial sparsity inherent in images, videos, and objects to reduce the amount of computation required for inference or training. For example, NSVF (Liu et al., 2020) constructs a sparse discrete representation using voxel octree. An implicit neural representation is then learned over the discrete set of voxels instead of the infinite set of coordinates. However, there are several drawbacks to these approaches. First, some of the mentioned approaches are only applicable to specific tasks, e.g., novel view synthesis (Neff al., 2021). Second, the additional data structures that are used explicitly save represen- tations to avoid computation may lead to high memory overheads, e.g., Garbin et al. (2021). Third, the effectiveness of these approaches is limited by the sparsity of the signal being learned and they may be less effective for highly dense and feature-rich images/objects.
Our goal, in this work, is accelerate both training and inference for a wide range of coordMLP- based representation tasks. To this end, we leverage the following observation. The inputs to these MLPs are typically multi-dimensional coordinate points, i.e., an (a,, y) coordinate for an image, (I, t) for video, etc., and in existing MLPs, each coordinate is processed independently, as depicted in Fig. (d). However, this misses the implicit locality among points in an image that have similar coordinate values, for example, the neighboring points along the same axis. In this work, we propose a new architecture where the initial layers are split to independently process each dimension of the input. Thus, each dimension is treated independently rather than each coordinate point itself. The outputs of these initial layers are then fused to generate a final prediction based on all input dimensions, as depieted in Fig. ((e). We refer to this split architecture CoordX.
The major benefits of our approach are: First, it enables significant speedups in both training and inference by essentially reducing the dimensions of the input. For example, the overall inputs of an H x W image fed into the MIP is reduced from H x W to H + W. At the same time, by leveraging the locality between points with similar coordinate values, we able achieve accuracy close to the baseline architecture. Second, this technique can be applied orthogonally to previously-propose optimization techniques that leverage sparsity without incurring much additional memory overheads. Third, the dimension-bar based splitting approach can be applied to wide range of tasks that use coordinate-based MLPs as we demonstrate in this work.
The key challenges in developing split architecture are: (i) ensuring that the overall size of the model does not increase even when processing each dimension of the input with a separate layer; and (ii) effectively fusing features from the split layers to retain the original model accuracy. In this work, we handle these challenges by (i) splitting only the first FC layer and then sharing the remaining FC layers across all input branches; and (ii) fusing features from split branches using an outer product operation. We demonstrate that CoordX can significantly accelerate training and inference for various signal fitting tasks' including image, video, and 3D shape fitting. Our analysis shows a speedup of ~2.5 using a 5-layer CoordX model for various tasks. We demonstrate a 2.35x speedup in inference time for 1K resolution image and a 2.78x speedup for 1 resolution video clip in our experiments.
To summarize, in this work, we make the following contributions:
We develop new architecture for coordinate- baadin MLPs that leverages locality between input coordinate points to reduce the amount of computation required and significantly speed up inference and training.
CoordX however, does not improve the performance when fitting unidimensional signals (c.g., audio).
2

---

Published as a conference paper at ICLR 2022
We design the split MLP architecture that achieves faster training and inference and re- tains the same level of accuracy as the baseline models with comparable or slightly larger parameter counts.
We demonstrate CoordX's effectiveness in speeding up inference and training various signal fitting tasks, including image, video and 3D shape rendering, and other advanced applications such image super-resolution.
2 RELATED WORK
Coordinate-based MLPs for implicit neural representation. Recent works have demonstrated the potential of MLPs in implicitly representing signals such as images (Stanley, 2007), volume occupancy (Peng et al., 2020; Mescheder et al., 2019), signed distance (SDF) (Park et al., 2019; Xu et al., 2019), texture (Saito et al., 2019; Henzler et al., 2020), and radiance fields (NeRF) (Mildenhall et al., 2020). Unlike MLPs used for other tasks, an MLP used for implicit representation needs to be able to capture the high frequency variations in the color or geometry of images and objects. To address this, Mildenhall al. (2020) proposes a positional encoding (PE) method that maps the original input coordinates into a high dimensional space using a series of high frequency functions to better capture high frequency variations. Sitzmann et al. (2020) use sinusoidal activation functions (SIREN) to replace the ReLU activations in the MLP, and Tancik et al. (2020) use simple Fourier feature mappings of input coordinates to learn high frequency variations. Many recent works also aim to make the signals represented by coordinate- based MLPs editable or generalizable. A common way to enrich the representation power of these models is to add latent codes/features as additional inputs to the coordMLPs (Schwarz et al., 2020; Park et al., 2021: Chen et al., 2021; Mehta et al., 2021). These additional inputs are combined with positional coordinates and are essentially treated as additional dimensions in the input coordinates. CoordX is directly applicable to these works.
Accelerating MLPs for implicit neural representation. Several recent works aim to accelerate inference and/or training for implicit neural representation using MLPs. There are broadly two directions among existing works: (i) reducing the number of points to be queried; (ii) reducing the time spent in processing each point. Redueing the number of query points is a common optimization method for ray-casting-ban based 3D neural representations. Due to spatial sparsity in a 3D scene, a significant number of sampled points along cast rays are empty, without useful density or color information. The inference/training speed can be greatly improved by skipping the processing of these points. Neff al. (2021) use an additional depth network to estimate object depth along each ray, hence only points around the object surface are sampled. Similarly, traditional explicit 3D representation methods can be leveraged to guide point sampling. Garbin et al. (2021) and Kellnhofer et al. (2021) explicitly extract the mesh of rendered object after training to reduce the number of points that need to be processed by the MLP. Liu et al. (2020); Yu et al. (2021): Takikawa et al. (2021) construct an octree structure to represent the 3D object during training that enables skipping empty spaces. To reduce the computation done for each point during inference, Reiser et al. (2021) replace the larger MLP for the whole scene with many tiny MLPs that have fewer layers and parameters, each of which only represents a small region of the whole scene. Garbin et al. (2021) and Yu et al. (2021) also pre-compute points and store results in memory. Therefore, only lightweight interpolation rather than intensive MLP computation is performed for each query point during rendering. To alleviate the significant memory requirements and processing time to pre-compute the original NeRF MLP's 5D input grid (3D position + 2D direction), these works leverage spherical harmonics (Ramamoorthi & Hanrahan, 2001) to generate a more efficient 3D representation.
Concurrent work ACORN (Martel et al., 2021) uses a multi-scale block-coordinate decomposition to efficiently represent large-scale images or 3D shapes in a hierarchical way. Unlike our coor- dinate decomposition, ACORN uses a quadtree or octree to decompose the input coordinate grid into small patches. ACORN first gets patch-level features via large coordinate-based MLP, and then uses these features to get predicted signals for each coordinate via feature interpolation and a lightweight MLP. Thus the number of queries to the large coordMLP in ACORN is significantly reduced. Though both ACORN and CoordX reduce the number of queries to some parts of the pardinaae-baseld model, they use very different approaches and these two approaches orthogo- nal. CoordX can be combined with ACORN to generate additional speedups by replacing ACORN's coordinate encoder with the CoordX model.
3

---

Published as a conference paper at ICLR 2022
W
[2]. [4]
[[]]
[c], (b),
X [[)) [2], [3],
H-
H
[3)
[0]]
x [[3), [2], [}) [M]]
Coordinate inputs X
(b) Decomposition
(c) Decomposed coordinates
KxM
III
[0,4]
[[1b"
Dd [I]]
Fuse
e
X(I
#x1
Doyb
NK2
[8,4]]
ILA,," by," [W]]
HxW
NXM
Coloryy
RXW x3 (ahtr reshapo)
Color... sy
(d) coordMLP
(e) coordX
Figure 1: (a)-(c) The process of decomposing coordinate inputs X for 2D image of size H X W into 2 split parts (1) and ( (d) Baseline coordMLP architecture. N = HW coordinate inputs (X) are independently fed into the MLP. (e) Our CoordX architecture. The initial layer is split up into separate branches for each decomposed coordinates ( and x After shared FC layers, the split features are fused and the fused features.
features are then fed into few FC layers to generate the final signal values. M is the size of the hidden
3 METHOD
3.1 NEURAL NETWORK ARCHITECTURE
In our proposed architecture, the initial layer of the MLP is split across C' input branches. The following D, 1 FC layers shared among the input branches. The fusion operation then fuses the intermediate features from the different feature branches and feeds fused features to last Dy FC layers generate the final signal values (see Fig. 1(e)).
Input decomposition along coordinate dimensions. In typical coordMLP models, a set of N coordinate points individually queried inputs, where N HW for image of size W in Fig. ((d). We refer to this set of input coordinate points as X E Nx where K is the dimension of each point (e.g.. K for (z, y) points on a pixel plane). With CoordX, instead of querying the first layer of the MLP with N coordinate points of dimension K, we split the first layer of the MLP into separate branches, where C S K. If K C, each dimension of the K-D input coordinate is separately fed to the corresponding C: input branch as shown in Fig 1(e). If C < K, in this case, or more coordinate dimensions are not split and retained their original form. For example, coordinate points in video (x, y, t) (K 3) can be decomposed along 2 dimensions (C' 2); branch fed by 2D coordinate points along the X and Y dimensions as (x, y) and branch is fed by the temporal dimension (t).
Thus the input grid X E R is transformed into set of decomposed input grids (x() B. = 2,,,..., C),, where is the input branch, B, is the number of coordinate points fed into branch and K, is the dimension of the each decomposed coordinate point fed into branch i. Thus, N -[= B, and K = K,.
First layer. The first layer of CoordX contains C parallel branches converting ( the corre- sponding hidden features H()) Each branch has a weight matrix w( E RKiMM where M is output feature size. With activation function Φi.), the first layer is 2:
[H ( (() -i= 1,2,...,
Layers before fusion. To avoid adding additional parameters to our model, we use the shared weight matrices (FC layers) to process hidden features from all parallel branches after the first layer (Fig. 1(e)). Suppose there are D2 layers before fusion, we can formulate these layers as:
w H = 1,2,...,C),foo 2...., D,
(2)
We omit the hias b in the following equations for readability.
4

---

Published as a conference paper at ICLR 2022
This design has the following benefits: (i) extra parameters added because the first layer split among the branches and the remaining layers are shared; (ii) higher computational effi- ciency/parallelism features from different branches can be concatenated and processed in one batch; (iii) sharing parameters between input branches enables capturing correlations between the different input dimensions.
Feature Fusion. The intermediate features H E R B.xMM from the branches are then combined to generate a fused feature H E IR Bi x..x X.
We use an outer product as the fusion operation (discussed in more detail 3.3). The fusion opera- tion is formulated as follows:
H = Fuse H 11) D.
HD.) = HD. ... . &H
(3)
Where & denotes the outer product operation. The last D, FC layers from the baseline model are retained to transform the fused features into final predictions.
Speedup Analysis. We assume each FC layer of the coordinate- based MLP has the same amount D-layer baseline coordMLP, there are coordinate inputs processed by each layer of the MLP. The resulting computation is of the order O(DN). In D-layer CoordX model with split branches, we assume B, (the number points processed by branch i) equals B, thus N = II B; = BC, Each of the C' branches from the first D, layers before fusion needs to process B decomposed inputs and the remaining D2 layers (D, + Dj D) process the fused features of the original size N. The two parts in CoordX together result in computation of the order OCCD,B N) The ratio y of the computational complexity between the two methods is
of computation.
DN = (D. (D++D))B 7 CD,B+ D CD, DG ACBI-C+1*
A-1
D. Dj
X ==
(4)
When B large enough, the term ACBI-C becomes negligibly small. Thus, the theoretical upper bound of the speedup that be achieved is - À + 1. A 5-layer CoordX MLP with D, 3 and D5 = 2 can thus achieve a 2.5x speedup over 5-layer baseline MLP based on our assumptions.
3.2 ACCELERATING TRAINING WITH COORDX
When input coordinates are split along each coordinate dimension, during fusion, the original coor- dinates are reconstructed by generating all combinations of the input values along each dimension. Thus, to perform this splitting for training, the original signal values must be available for all combi- nations of the split inputs. For example, if the pixel points [(1, 1), (2, 2), (3, 3). (4, 4)] are sampled, they cannot be split into two input vectors for each coordinate dimension, because some combina- tions of the input vectors are missing (c.g., (1, 3), (1, .)). If points [(1, 1), (1, 2), (2, 1), (2, sampled, we can then split them into two [(1), (2)] vectors along each dimension. In a typical training process, points are randomly sampled and all combinations will not available.
To address this, first sample positions along each decomposed dimension, then these sam- pled positions to generate all combinations of coordinate points (see Fig. 4). The number of sampled positions along each dimension is approximately proportional to the dimension length that each separate branch has sufficient number of decomposed coordinates to learn while the uniform dis- tribution of the sampled points is still guaranteed. We discuss how this is done in more detail in Appendix A.
3.3
THE SIGNAL DECOMPOSITION OF COORDX.
Signals such as images can be decomposed into multiple matrices using mathematical methods. For example, a gray-scale image (represented as a 2D matrix) can decomposed into two matrices using an SVD thought of as learning multiple matrices that are then used to reconstruct the original signal, as depicted in Fig. 2. By moving the fusion operation (outer product) to the end of the model (D, 0), we get a fully split CoordX model. This fully split CoordX model is also able to represent the image with high PSNR (Fig. 2(b)), indicating the ability of the model to learn decomposed signals.
QR decomposition. Similarly,
proposed split architecture essentially be
5

---

Published as a conference paper at ICLR 2022
HKR H
#x1
Out
Hy
Wx1
WXR
(a) Fully split CoordX
Decomposed image matrices
Figure 2: CoordX learns the image decomposition. (a) The fully split coordX architecture. The fusion operator directly generates the predicted signal. It is the dimension reduced by the summation after the outer product. (b) The image can represented as matrix multiplication between two learned feature tensors (matrices) H, and H.. 256, = 512 this example.
A key difference between the previously-presse presented CoordX model (c.g.. Fig. I(e)) and the fully split CoordX model (e.g., Fig. 2(a)) is the location of the fusion operation. Our fusion operation defined in Eqn. 13 is series of outer products, while the fully split CoordX in Fig. 2 requires an outer product to generate the final prediction without an additional FC layer. In order include these two cases in one fusion equation, we extend the dimension of the split features to be fused. Let the hidden features after D, layers are of feature size M. The M hidden feature values then can be reshaped into R x S matrix (where M RS). R is the size of the dimension reduced by the summation after outer product, and S represents the feature/signal size after the fusion. For example, R 1,5 M for fusion in the partially-split CoordX. The fully split CoordX in Fig- 2 has R = 256 and 5 = 1 (single channel image). Following our definition 3.1, each branch's hidden feature H()) E RBxxM before the fusion can be reshaped into (() E BBxSS The fusion then can be formulated as
H PC
H
(5)
p; is positional index for feature tensors, i is the index for dimension R of (1) H E B. x... xBe B8x The fusion operation defined in Eqn. 13 special case of Eqn. 5 when R 1. While R = 1 is sufficient to learn an accurate representation for most signals, if we set R > 1 (e.g., R 2 or for the hidden feature fusion and enlarge the corresponding FC layers by scale factor R to keep S unchanged, the representation accuracy of the CoordX model can be improved (evaluated Sec. 4.4).
4 EXPERIMENTAL EVALUATION
In this section, we evaluate the performance and efficiency of CoordX for several signal fitting tasks, including images, videos and 3D shapes. We also evaluate CoordX for latent code coordMLPs.
Setup. We choose several existing coordMLP models as baseline models for different tasks (details are in the following subsections). All models are implemented on Py Torch (Paszke et al., 2019) and evaluated using an NVIDIA RTX3090 GPU. We use SIREN to denote models with periodic activa- tion functions (Sitzmann et al., 2020), and we use PE to denote models using positional encoding with ReLU actívation (Mildenhall et al., 2020; Tancik et al., 2020). Unless otherwise specified, our experiments use 5-layer MLPs. The CoordX models have two FC layers after the fusion operation (D,- = 2). Model hyperparameters such learning rate, batch size, number of epochs, etc., are the same as those used the corresponding baseline coordMLPs. The output image/object visualiza- tions and qualitative comparisons are in Appendix B.
4.1 SIGNAL FITTING WITH COORDX
Image Representation. In this experiment, 5-layer coordMLPs with 256 hidden units are used to fit RGB images of size 512 x 512. We randomly select 12 center-cropped images from DIV2K dataset (Agustsson & Timofte, 2017) to report the average PSNR. Both SIREN and PE models are tested. Table shows the PSNR and the required training time for 20,000 epochs. Fig. 3(a) compares the inference speed between the baseline SIREN model and CoordX for different image sizes.

---

Published as a conference paper at ICLR 2022
NLP Cur
200.00
0.4
70112
702
140.00
250
522
662
403
00.00
302
20.00
66.00
14.4
100
21.0
29:3 28
100
0.00
250x
512-532
2048-2348
2b6
>32
1004
12813
Resollation on
(a) Image inference
(b) Video inference
(c) Shape inference
Figure 3: Comparing inference speed. (a) average inference time for the SIREN model for a synthetic image
on different resolutions: (b) millisecond per frame to represent a synthetic video with 100 frames on different resolutions with PE models; (c) inference time of a grid of coordinate points for marching cubes to display 3D
shapes; SIREN models are used.
Method
MLP-S
CoordX-S
CoordX-S-A
MLP-P
CoordX-P
CoordX-P-A
PSNR
40.25
37.08
37.02
37,30
33.89
33.84
Tonining (min)
12.07
22.33
5.73
14.80
27.08
5,06
Table 1: Accuracy and speed comparison. MLP indicates the baseline coordMLP. S and P refer to SIREN and PE respectively. A refers to accelerated training with our sampling method (Sec. 3.2).
We make the following observations. First, CoordX can achieve high PSNR values similar to those achieved by the corresponding baseline coordMLPs, with a 3 dB drop in PSNR (we discuss ways to alleviate this in 4.4). Second, training the CoordX model is almost twice as fast as training the baseline coordMLP. Third, CoordX achieves significantly faster inference times compared to the baseline (up to 2.83x for 2K image). The observed speedups are higher for bigger images.
Video Representation. In this task, we use 5-layer coordMLPs with 1024 hidden units to learn 10-second video clips. We evaluate video clips. The baseline MLPs are described in Appendix B.
For the CoordX model, we consider two splitting strategies. The first one is to split the input coordi- nates into 3 vectors along the height, width, and time axes, resulting in 3 network branches (referred to as X+Y+T). The second strategy splits the input coordinates along the spatial and temporal di- mensions (2 branches). The height and width dimensions are flattened into one input branch and the time dimension is the second branch. We refer to this strategy as XY+T. We use the PE method in this experiment as it performs better than SIREN for videos. Table shows the average PSNR and the required training time for 100,000 epochs. Fig. 3(b) compares the inference speed between the baseline model and CoordX for different frame resolutions.
We make the following observations. First, while both X+Y+T and XY+T splitting strategies achieve high PSNR, XY+T performs slightly better than X+Y+T and even outperforms the base- line model. Second, training CoordX with our sampling strategy (XY+T-A) is 2.02X faster than training the baseline model while retaining high representation quality in the model. Third, CoordX achieves 2.78x speedup over the baseline MLP for inference, and the speedups are higher for higher video resolutions.
3D shape representation. We choose occupancy networks (Mescheder et al., 2019) to represent 3D shapes. The output probabilities indicate whether query points are inside or outside the shape. We use 5-layer coordMLPs with 256 hidden units for this experiment and the SIREN method it performs better than PR.
We use loU as the quality metric (see details in Appendix B) and report the average IoU obtained for 4 3D objects. Table 2 shows the average IoU values and the required training time for 10,000 epochs. To display shapes and compare inference speeds after training, we use the marching cube' (Lorensen & Cline, 1987) process in which 3D voxel grid needs to be queried. Fig. 3(C) compares the inference speed for different grid resolutions.
We make the following observations. First, we achieve similar IoU scores with CoordX compared to the baseline coordMLPs. Second, CoordX is up to 1.7x faster for training speedup compared the baseline model. Third, CoordX achieve speedup of up to 2.81x over the baseline for a resolution of 256 and on average 2.55x.
Voouuee rendering, as another way to display 3D shapes, will be discussed in Appendix C.
7

---

Published as a conference paper at ICLR 2022
Task
Video
3D shape
Method
MLP
X+Y+T
XY+T
XY+T-A
MLP
CoordX
CoordX-A
Metrie
33.315
31.69
33.63
33.35
0.991 / 0,970
0.991 / 0.970
0.989 / 0,964
T Lnnining
4.30
6.63
5.22 h
2.13
5.72 min
16.14 min
3.28 min
Table 2: Accuracy and speed comparison for MLP training. MLP indicates the baseline coordMLP. A refers to training CoordX with our sampling strategy. Video task uses PSNR as the metric, and 3D shape task uses easy/ / hard IoU (higher is better) as metries.
In summary, we demonstrate that the CoordX models are able to provide significant speedups in both training and inference over baseline models for different sets of tasks. We also demonstrate that the CoordX models are able to retain the representation quality of the baseline models.
Novel View Synthesis. CoordX can also be used to accelerate NeRF-like volume rendering (Milden- hall et al., 2020). However, NeRF has an irregular input coordinate grid (see Fig. 8), and thus CoordX cannot be directly applied here. We instead use a simple mechanism in C.4 to accelerate grid-interpolation-aan
NeRF rendering with CoordX. This mechanism involves the precomputa- tion of densely sampled coordinate points and grid interpolation over the MLP predictions. Our proposed method can achieve a 1,81x Ix training speedup and 4.41x rendering speedup over baseline NeRF with a quality drop of less than 0.5 dB on low-resolution blender scenes.
4.2 COORDX WITH LATENT CODES
In this part, we test our method on more complex coordinate- MLPs, which include high- dimensional latent codes a part of the model input for better generalizability. The model we accelerate is LIIF (Local Implicit Image Function) (Chen et al., 2021) for image super-resolution. It takes 2D pixel position and high-dimensional LIIF latent code from a CNN encoder as inputs and then predicts the RGB value the corresponding pixel coordinate. The coordMLP (decoder) here a 5-layer MLP with 256 hidden units. We use a CoordX model with 4 branches (code.x.y.cell) to replace the baseline coordMLP for this task, where cell is an additional 2D input introduced in Chen et al. (2021) for better accuracy. We use a pre-trained EDSR (Lim et al., 2017) decoder and train our CoordX decoder from scratch for 1,000 epochs. We use the DIV2K dataset (Agustsson & Timofte, 2017) for quantitative comparisons. Table 3 compares the baseline LIIF and our modified CoordX LIIF on different up-sampling scales, including scales used in training and not used in training.
We observe that the CoordX LIIF model achieves almost the same PSNR compared the original baseline model. At the same time, CoordX achieves significant speedups (up to 2.88x) for inference for a target 2K image super-resolution. Since the MLP only forms part of the overall training pipeline, CoordX does not generate significant speedups for training. We conclude that CoordX is more generally applicable to variations and enhancements of coordinate-based MLPs.
Method
In-distribution
Out-of-distribuue
Speed
Up-sampling scales
x2
x3
x4
x6
x12
x18
x24
x30
(x2)
MaSSR--haelline-.It
34.67
30.96
29.00
26.75
23.71
22.17
21.18
20.48
131.03 ms
EDSR CoordX-LLi
34.65
30.95
28.99
26.67
23.60
22.08
21.10
20.42
45.53 ms
Accuracy and speed comparison for MLP inference. PSNR is used as the quantitative metric. (Scores for DASS-baaeline-LIIF are from the original paper.) The speed here is the average time to finish processing all images in the DIV2K valid set with x2 scale.
Table 3:
4.3 THE DEGREE OF SPLITTING FOR MLP
CoordX models can be partly split or fully split (Sec. 3.3) and in this section, we evaluate the impact of different degrees of splitting for 3D shape fitting. We treat the number of layers after fusion as a hyperparameter (D₂) to be swept. We vary D, from 0 to 5 for the 5-layer CoordX models evaluated. Note that when D, 5 the coordX model is essentially baseline coordMLP. To demonstrate that similar representation accuracies as CoordX cannot be obtained by simply using MLPs with fewer layers, we also evaluate baseline coordMLPs with different model depth (DmLe). All CoordX models use our sampling strategy (Sec. 3.2) during training. We use a 1283 grid of coordinate points to test the inference speed. The test results are listed in Table 4.
8

---

Published as a conference paper at ICLR 2022
Dj
DMLP
Model setting
5*
4
3
4
IoU
0.976
0.980
0.974
0.963
0,941
0.880
0.959
0.876
0.516
T Ininnng (min)
5.72
5.40
4.24
3.28
2.27
2.38
4.44
3.11
1.67
T inference (ms)
105.22
95.43
69.37
46.25
19.73
22.18
82.12
55.21
30.77
Table 4: Accuracy and speed comparison. When D, 5, CoordX becomes the baseline coordMLP.
We make the following observations. First, as we have more split layers (D3 is lower), the IoU score decreases slightly. This indicates tradeoff between representation accuracy and training/inference speed. Second, from our comparison with smaller baseline MLP models, we observe that similar representation accuracies cannot be obtained by simply using fewer layers (which may be faster).
4.4 IMPROVING REPRESENTATION QUALITY
As observed in Section 4.1, CoordX leads to significant speedups but causes drop in accuracy (~3dB PNSR) specifically for image representation. Compared to other types of signals, image signals typically have more high frequency variations. The decomposition of inputs in CoordX may lead to loss of information in capturing these high frequency signal variations.
To improve the representation quality of CoordX as discussed in Section 3.3, we increase the size R of the reduction dimension (R = for models evaluated previously) and keep the fused feature size S unchanged (S 256 for image models). We increase R to 2 and in this evaluation and extra parameters are thus added to the splitting FC layers. This can be done in two ways. First, we can increase the number of hidden units the layer just before fusion by a factor of R (referred as +). A Alternatively, we can increase the feature size of each splitting layer before fusion by a factor of R (referred to as ++). This approach requires the use of more parameters.
We use SIREN models for this experiment and the experimental settings are the same in 4.1. Table 5 compares the different model confi gurationss...
Model
MLP
CoordX
R2+
R2++
R3+
R3-+
PSNR
40.25
37.08
39.70
41.32
41.60
43.55
Training time (min)
12.60
5.87
7.67
7.87
9,00
9.33
Inference speed (ms)
52.45
22.34
29.24
34.92
30.05
37.73
Model size (MB)
0.77
0.77
1.02
2.27
1.27
4.78
Table 5:
Image fitting with larger CoordX models.
The inference speed is the average time to
inference a synthetic 1024x 1024 image 100 times.
From Table 5, make the following observations. First, increasing R can effectively improve the representation quality. The augmented CoordX models are also able to achieve higher PSNR than the bascline coordMLPs. Second, even though more parameters are added to the augmented CoordX models, the training and inference speeds are still faster than the baseline coordMLP. Third, for the same R, the R++ strategy which requires more parameters is more effective. However, increasing the R value is more beneficial compared to using more parameters with the R++ strategy.
CONCLUSION AND FUTURE WORK
In this work, we propose CoordX, a new architecture accelerating the inference and training of coordinate-based MLPs for implicit neural representations. CoordX decomposes input positional(- temporal) coordinates along each input dimension and splits the baseline MLP into a multi-branch MLP. Decomposed inputs separately feed into each branch leading significant speedups in in- ference and training while retaining similar representation accuracy as the slower baseline models. We also demonstrate approaches to increase the representation capacity of split coordinate-baser MLP, leading to new design space for trading off speed, quality and size.
CoordX also has wider applicability beyond the tasks evaluated. For example, the splitting strategy introduced in this work can also be applied to accelerate view synthesis for dynamic scenes (Peng et al., 2021; Li et al., 2021), and deformable scenes (Park et al., 2020: 2021). CoordX can also help neural implicit representation based compression (Dupont et al., 2021) achieve higher com- pression/decompression speed. Lastly, the split MLPs proposed in this work can potentially be used for dulti--aimesioonll signal decomposition multivariate function decomposition, which may be useful for other tasks such as multi-relational data modeling (Rabanser et al., 2017). We leave these explorations future work.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank Chen Yang, Zanwei Zhou for interesting discussions throughout the project. We also thank Gavin Guan, Jimmy Lin, Sankeerth Durvasula well anonymous reviewers for helpful comments and suggestions of this paper, Bin Ji, Shunyu Yao, Dennis Huang from VokaTech, Shanghai Jiao Tong University for computing support.
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference computer vision and pattern recog- nition workshops, pp. 126-135, 2017.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8628 8638, 2021.
Emilien Dupont, Adam Goliñski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Com- pression with implicit neural representations. arXiv preprint /0xiv:2/03.03/23, 2021.
Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. arXiv preprint arXiv:2103.10330 2021.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometrie regu- larization for learning shapes. arXiv preprint arXiv:2002. 10099, 2020.
Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Learning neural 3d texture space from 2d exemplars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. $356-8364, 2020.
James T Kajiya and Brian Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics, 18(3): 165-174, 1984.
Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neu- ral lumigraph rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4287-4297, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang Lv. Neural 3d video synthe- sis. arXiv preprint arXiv:2103.02599 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolun resolutono Proceedings of the IEEE conference on com- puter vision and pattern recognition workshops, pp. 136-144, 2017.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. arXiv preprini arXiv:2007. 15571, 2020.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169 1987.
Julien NP Martel, David B Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:2105.02288. 2021.
Ricardo Martin-Bruall Noha Radwan, Mehdi SM Sajjadi. Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collec- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7210-7219, 2021.
10

---

Published as a conference paper at ICLR 2022
Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. arXiv preprint wxXh::2104003960, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks. arXiv preprint arXiv:2103.03231 2021.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 1453--11464, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan Goldman, Steven M Seitz, and Ricardo MaartnnBrualle Deformable neural radiance fields. arXiv preprint arXiv:201 1, 129448, 2020.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo -autin-Brualla,, and Steven M Seitz. Hypemerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106. 13228, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Sida Peng, Yuanqing Zhang. Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou, Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9054-9063, 2021.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 523-540. Springer, 2020.
Stephan Rabanser, Oleksandr Shchur, and Stephan Günnemann. Introduction to tensor decomposi- tions and their applications in machine learning. arXiv preprint arXiv: 10088, 2017.
Ravi Ramamoorthi and Pat Hanrahan. An efficient representation for irradiance environment maps. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp. 497-500, 2001.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. arXiv preprint 1xXiv22103.13744, 2021.
Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implieit function for high-resolution clothed human digitization. In Proceed- ings of the IEÉE/CVF International Conference on Computer Vision, pp. 2304-2311 2019.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. arXiv preprint arXiv:2007.002422 2020.

---

Published as a conference paper at ICLR 2022
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Kenneth O Stanley.
Compositional pattern producing networks: novel abstraction of development. Genetic programming and evolvable machines, 8(2):131-162, 2007.
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11388-11367, 2021.
Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006. 2020.
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint a9Xiv: 10711, 2019.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. arXāv preprint arXiv22033.10024 2021.
12

---

Published as a conference paper at ICLR 2022
A INPUT POINT SAMPLING FOR COORDX TRAINING
Randomly sampling points for MLP training will not lead to any speedups with CoordX. This is be- cause randomly chosen coordinates cannot be decomposed into input vectors along each dimension, as discussed in 3.2.
Thus, need coordinate dimension. To ensure this, we need to select all input points that are combinations of the split imput vectors. To this end, we formulate simple sampling strategy that describe below.
sampling strategy that ensures that the chosen points be split along each
Suppose we need to sample points to learn the signal bounded in a C-D coordinate space, where each dimension i is of length S,- We assume that the number of sampled positions along each coordinate dimension is approximately proportional to the dimension length S. so we can represent the number of sampled positions along each dimension pS; where 1 scale factor. Because we require original sampling and our new sampling should have approximately the same number of sampled points in the same setting, we have:
Il U.. N
(6)
By solving this equation, can get
N V
V
I S
Though we limit the flexibility of the sampling process, the probability of choosing a particular point/region is the same for two sampling methods. To introduce more randomness to our sampling method, we also add random noise to each J.S,, Note that the sampled coordinate space can be continuous or discrete, but this split sampling method works in the same way.
As result, this strategy ensures that the points selected can be fully decomposed along any dimen- sion. All input points that are formed when the coordinates are fused are thus selected for training. This is depicted in Fig. 4.
S
nS1
S2
8-+
2D Sample Space
Sampled Points
Figure 4: An illustration of our point sampling strategy in 2-D space. The data scale is S S2 on the first and second dimensions, our new sampling strategy ensures that the proportions of samples along each coordinate dimension are exactly the same and can be split along each dimension.
B ADDITIONAL TASK DETAILS
In this section, provide additional details and results regarding various fitting tasks evaluated in Section 4. All experiments on signal fitting tasks are implemented using PyTorch and use an NVIDIA RTX3090 GPU. We the Adam optimizer (Kingma & Ba, 2014) during training.
B. IMAGE FITTING
This task involves training the model to fit 2D images. Given a 2D pixel coordinate, the model predicts the color of the pixel that coordinate. One of the biggest advantages of using implicit
13

---

Published as a conference paper at ICLR 2022
representations is that it can represent images with an arbitrary resolution, since it learns contin- uous representation of the image, therefore can represent an image at high resolution but with a size smaller than the image itself. We formulate this task as:
T IR2 + T
(8)
Where O is the dimension of the pixel color value of the image, for example O = 3 if image is represented in RGB value, or O if greyscale image.
In this experiment, we try to fit RGB images of size 512 x 512. We randomly select 12 center- cropped images from DIV2K dataset (Agustsson & Timofte, 2017) for this experiment, and train for 20,000 epochs.
In the image fitting tasks using PE, positional encoding maps input point x E I& that is normalized to the range [-1, to R 2kd using set of sinusoidal functions:
(9) Where K is the dimension of each input point, p an individual coordinate in input point x, and d is the encoding frequency. We then concatenate this encoding with the original input x E RK to get an input of dimension 2Kd d'd + K. We follow the settings and the encoding frequency used in the image fitting task introduced in (Tancik et al., 2020).
<p) (sin(2" TT), cos(2"Tp))...... ..., (i, xp). (2) cos
xp))
We separately sample 2D pixel coordinates each axis. We sample 512 points from both axes for each training batch. Since the image size is 512 x 512, we can fit all sampled points into one batch on an RTX3090 GPU.
In Fig 5, we choose images of various complexities from the DIV2K dataset to show the qualitative results for the image fitting task.
B.2 VIDEO FITTING
We also train our model to learn video representations. Compared to images, videos have an extra dimension which the temporal dimension or the frame number. Therefore this fitting task has an input of 3D coordinates (z, t) where (z, y) is the pixel spatial location, and (t) is the temporal
location of that pixel, or the frame number it at. Similar to the image regression task, we formulate this task as:
(10) Where O is the dimension of the pixel color value of the video at the given pixel location at the given frame. We also supervise the model using MSE loss between the predicted color and ground truth color. There are two splitting strategies. In the first strategy, we split all three dimensions of the input coordinates, for a set of coordinates in the format (z, y, t), we split into (z). (3) and (t) and sample values in each separated channel. We denote this strategy as X+Y+T. In the second strategy, we only separate the pixel/spatial dimension with the frame/temporal dimension, for set of coordinates in the format (I, t), we split it into (x. y) and (t) and sample values in each separated channel. We denote this strategy XY+T.
$ R
In this experiment, we use 5-layer coordMI.P Ps with 1024 hidden units to fit 2 10-second video clips", and we train for 100,000 epochs.
While performing our sampling strategy for CoordX with XY+T splitting mentioned in Sec. 4.1, for each training batch, we first randomly permute frame values and choose the first values to be the frames to sample. We sample 3.8% of the total points each batch, therefore for each frame, we sample y = 0.038N/t points (the number is rounded to the nearest integer), where N denotes the total number of points in the 3D space for the video. The coordinate values along the spatial (XY) dimension and the chosen frame values along the temporal (T) dimension for each sampled point are fed to CoordX. The sampling method for the X+Y+T strategy is very much similar, except each chosen point further split along the XY dimension. Like the image fitting task, the model is supervised using an MSE loss of predicted color and ground truth color for each pixel.
We provide the qualitative results for the video fitting task in Fig 6 and 7, where we capture and show several frames of the two videos for comparison.
A bike video from http: / //www www.ww...kk...ooo. vititiiiieee..ee orreeeorgg//ae /tagg/ttabl... e/ e/ e/dataseta html, and cat video from https: / /www. pexe a. com
14

---

Published as a conference paper at ICLR 2022
(a) X+Y, SIREN
(b) SIREN
(c) X+Y, PE
(d) PE
(e) Ground Truth
Figure 5: Qualitative results for the image fitting task. The 4 images shown are from the DIV2K
dataset, center-cropped and down-sampled 512x112 resolution.
B.3 3D MODEL FITTING
We represent 3D shapes using a ID occupancy grid (Mescheder al., 2019), inspired by similar works in (Tancik et al., 2020). An occupancy grid represents the shape as a ID occupancy probability for given 3D coordinate and indicate whether the query points are inside or outside the 3D shape:
Ø D:R R
(1I)
For this task, we split the input coordinates along all three dimensions. We sample values along each of the three coordinate channels of the input 3D coordinates. We use 5-layer CoordMLPs with 256 hidden units to 3D objects (Bunny, Armadillo, Dragon and Buddha) from The Stanford 3D Scanning Repository". and train each for 10,000 epochs. We follow the settings in Tancik et al. (2020), and use cross-entropy loss for the model to learn the classification of occupancy labels.
We first recenter points loaded from mesh points [-1, 1] for SIREN test. For each batch, randomly choose number of points to sample per axis. We store those sampled points and their corresponding ground truth occupancy value file, to skip the consuminn data sampling step, which acts as major performance bottleneck in the training process, in order to present more accurate results of our split acceleration.
15

---

Published as a conference paper at ICLR 2022
(a) Ground Truth
(b) PE MLP
(c) X+Y+T
(d) XY+T
(e) XY+T with split acceleration
Figure 6: Qualitative results for the video fitting task (the cat video).
(a) MLP
(b) X+Y+T
(c) XY+T
(d) XY+T with split acceleration
(e) Ground Truth
Figure Qualitative results for the video fitting task (the bike video).
16

---

Published as a conference paper at ICLR 2022
B.4 FEATURE FUSION
In 3.1 while we choose multiplicative modulation (outer product) as our feature fusion strategy. here we present comparison between our approach against additive modulation (summation) and concatenation.
Assuming fusion occurs after the D,-th layer and we split the input into C' branches, additive modu- lation and concatenation strategy first reshape each feature H E B from branch i to become H D. x.xxx..BBB.bbb by expanding its dimension and repeating the feature along each new di- mension with corresponding number of times. The fused feature for additive modulation strategy can then formulated as follow:
H
D.
Σ
H D. ==1
(12)
Where H E RP x.. BexM M
the fused feature generated by additive modulation.
The fused feature for concatenation strategy can be formulated follow:
Cat D,
H 11 D, + ...
(C) D,
(13)
Where @ denotes the concatenation between feature tensors along the last dimension, and H
RB Bi x Be (CM) is the fused feature generated by concatenation.
The multiplicative and additive modulation doesn't 't require any modification to the FC layers after fusion, we simply retain the FC layers from the baseline model. Concatenation strategy requires modification to the first layer after fusion. We denote that layer as the ketth layer which originally has the formulation (He <(W; } where X E R B1 x.. Be xxxxxxxxxxxxxxx W E R M x and Mout is
equal to the number of hidden features in the subsequent layer, or the fitted signal dimension k-th layer is the last layer in the network. Since input feature now has last dimension size CM instead of M, the k-th layer is modified as {H's = of (W' xX'') where X' E R Bg x- x Be XCM W' e R RCM Moet, This modification also increases the number of parameters in the k-th layer.
We follow the same experimental settings in 4.1 and perform image fitting tasks training for 10,000 epochs using the additive modulation, multiplicative modulation, and concatenation fusion strate- gies. Results are shown in Table 6. We observe that multiplicative modulation performs the best out of the three strategies. Additive modulation and multiplicative modulation the same number of parameters, while concatenation requires more parameters than multiplicative modulation. This results in ~8% increase in training time compared multiplicative modulation.
Method PSNR
Prod-S
Sum-S
Concat-S
Prod-P
Sum-P
Concat-P 28.18
36.32
33.03
35.10
31.41
26.51
Table 6: Quality comparison between different fusion strategies. Prod, and concat denote multiplicative modulation, additive modulation and concatenation. and P refer to SIREN and PE respectively.
C VOLUME RENDERING WITH COORDX
Volume rendering is another important application of coordinate-baat based MLPs in 3D graphics visu- alization. Unlike the previously discussed tasks which show good input data alignment (i.e., the input coordinates can be easily decomposed along each dimension), the volume rendering applica- tion does not have this property in its sampled coordinates, e.g., Fig. 8. To enable CoordX's use in volume rendering tasks, into axis- axis-aligned grids. The details of our solution will be discussed in the following sections.
interpolation method that transforms the point sampling space
C.I VOLUME RENDERING
NeRF (Mildenhall et al., 2020) performs ray rendering during training and inference. It essentially renders 2D image based the camera ray from given camera location. This method is derived
17

---

Published as a conference paper at ICLR 2022
Figure 8: Illustration of sampled coordinate points for volume rendering.
from the classic volume rendering technique (Kajiya & Von Herzen, 1984) render 2D image given the camera location. The expected color (rr) where r(t) = o td is the function representing ray from camera, with near and far bound En ty can be described as:
c(r)= = 1'' T()))(((())(((()) dydt where T(t) = exp( S( (r((s))ss)
(14)
in
To estimate this continuous integral shown in the Eqn. 14, NeRF uses volume sampling technique, where 3D points are densely sampled along each camera ray, the color and volume density at each point is accumulated and used predict the final color of the camera ray in order to render the final image:
i-1 E
C(r) ~ Σ T()) - -
exp((o,,p) where T(i) = exp(
(15)
j=1
Here 6; is the distance between adjacent point samples 4, 4-14,1 and a; the color and volume density of the sampled point.
In real applications, hierarchical sampling is used for higher-quality rendering, A coarse sampling first performed the rough depth information of the object can be obtained. Next, do fine- grained sampling around the object surface based estimated depth information.
C.2 RAY SAMPLING WITH A SPLIT MLP ARCHITECTURE
A2- 2
Shape
Query the MLP
Output
Occupancy
Ke
Grespseyyt PuAk dicupoootta//..
Ray Casting
Sampled
Occupancy
Inlinear Interpolations
Output
Figure 9: Illustration of our modified volumetric rendering.
As explained in C.I. volume rendering includes casting a ray from the camera to each pixel of the rendered image, and sampled points along that ray. After that, each sampled coordinate point
18

---

Published as a conference paper at ICLR 2022
is used as the imput to the coordMLP, and it predicts a signal for each point. These predictions are then accumulated with other points on the same ray using rendering functions to generate the final color of the ray at that pixel. The difficulty of incorporating our CoordX model with volume rendering is that the sampled points are not axis-aligned and cannot be decomposed along coordinate dimensions. Since the camera ray is usually not aligned with any of the x; y, 2 axis in 31D space, sampled points along the ray do not share coordinate values (Fig, 8). To address this problem, we use an interpolation-baase rendering method that outputs predicted signal values of queried points by interpolating on a pre-computed grid. The interpolation- -based acceleration can also be seen in other relevant work such as FastNeRF (Garbin et al., 2021) and PlenOctrees (Yu et al., 2021).
As illustrated in Fig. 9. we first pre-compute 3D occupancy grid for the 3D object with pre- defined resolution (e.g., 256). Each voxel inside the grid occupies small cubic space. After pre-computation, if we are to render an image from an arbitrary camera angle, we can perform interpolation- based volumetric rendering, where the occupancy of the points we sampled along each ray is trilinearly interpolated from the 8 endpoints of the voxel cube which this point is in. We then get accumulated color/depth for each ray using the original volumetric rendering methodology.
Based on this interpolation raycasting method, we can replace the baseline CoordMLP with our CoordX model to accelerate the grid pre-computation step (the most consucinu step in the whole processing pipeline). The points in the pre-computed 3D coordinate grid can be decomposed into 3 vectors which are used to query CoordX. We demonstrate 2.8x speedup with CoordX in Fig. 3(c).
A similar approach can be used to accelerate training for NeRF models. However, using grid inter- polation for training requires training with all the points in the grid, which can be much larger than the number of points required to train the baseline model. Thus our grid interpolation-based training may consume more memory and take longer training times. To address this problem, we decide to first compute the split features just before fusion for the corresponding grid. For each query point, bilinear interpolation is performed directly on these split features, the interpolated features will then be fed into the second part of our splitting model to predict final signal values. In this way, we strike a balance between grid reuse and computation efficiency.
C.3 VISUALIZATION OF LEARNED 3D OCCUPANCY.
We apply our proposed rendering method to MLP models for 3D occupancy representations trained in 4.1. We choose the original volumetric rendering method as the baseline rendering method. We choose 512 x 512 as the target resolution of the rendered images. We uniformly sample 256 points along the ray for each level of two granularity levels in hierarchical rendering. The coordinate grid used our grid interpolation rendering method has the resolution of 384 for the longest length of the object's 3D bounding box. Fig. 9 shows the qualitative comparison of the rendered images. Our grid interpolation based rendering is referred to as grid in the figure. Table shows the speed comparison of different rendering methods. We test the time required to render one image (including the time to pre-compute the grid for our method).
We make the following observations. First, grid interpolation rendering can achieve significant speedups compared to baseline volumetric rendering (2.70x and 14.83x for baseline and CoordX respectively). Second, with grid interpolation rendering, CoordX achieves a 2.15x speedup over the baseline MLP. Third, the rendered images in Fig. 9 show our rendering method is able to render high-quality images for different ML. models.
Model
baseline MLP grid
CoordX
Rendering Method
baseline
baseline
grid
T Tendering (sec)
6.61
2.45
16.91
1.14
Table 7: Rendering time comparison for different volumetric rendering method.
C.4 NOVEL VIEW SYNTHESIS
We train a simplified 8-layer NeRF model for three blender scenes (Lego, ship and drums). We use the spherical harmonics (SH) function proposed in Yu et al. (2021) decompose the point position
19

---

Published as a conference paper at ICLR 2022
and view direction that the input the NeRF MLP is just the positional coordinates (z, y, 2). We refer to this model as our baseline NeRF model. Our CoordX NeRF model is based on this baseline model. We split the positional coordinates (z, y, 2) in branches and FC layers kept after the fusion operation (D; = 6, D3 = 2). The resolution of the feature grid used for bilinear interpolation set to 384.
To train these NeRF models, we use downsampled ground truth images (400 x 400) for supervised learning. 128 points sampled along randomly selected rays. For simplicity, do not hier- archical sampling for NeRF models. The models are trained for 200,000 epochs. To visualize the scene learned by NeRF models, we evaluated both baseline volumetric rendering and grid interpo- lation rendering (similar to 3D shape rendering). The target resolution of the rendered images is 400 400. The coordinate grid used has a resolution of 384 for the longest length of the object's 3D bounding box. We sample 256 points along the ray (without hierarchical sampling). The PSNR evaluated based on these rendered images. Table 8 compares the baseline NeRF and CoordX NeRF from different aspects, and Fig. 11 and Fig. 12 shows the rendered images for qualitative comparison.
PSNR
Metrics
T Trandeing (sec)
Tuining (hr)
Lego
Ship
Drums
baseline
29.65
28.28
24.77
3.09
baseline-NeRF
4.73
grid
29.23
27.93
24.61
1.97
baseline
29.29
28.05
24.84
7.82
CoordX-NeRF
2.61
grid
29.29
27.75
24.76
0.70
Table 8: Quantitative comparison between baseline NeRF and our CoordX NeRF. The speeds in the table tested on the Lego scene.
We make the following observations. First, our proposed interpolation-basee training can help CoordX achieve 1.81x training speedup over baseline NeRF. Second, our CoordX NeRF model achieves similar PSRN scores (less than 0.5 dB drop) compared to the baseline NeRF Third, our grid interpolation rendering method can achieve the same level of rendering accuracy (less than 0.5 dB drop) compared to the original rendering method for both baseline and CoordX models.
In summary, demonstrate that our proposed grid interpolation method is able render images with comparable quality with significant rendering speedups. This method also enables our CoordX model achieve training and inference acceleration over the baseline model.
20

---

Published as a conference paper at ICLR 2022
(a) MLP bascline
(b) MLP grid
(c) CoordX grid
(d) CoordX-A grid
(c) Ground Truth
Figure 10: Images result rendered using different ray rendering methods.
(a) NeRF baseline
(b) NeRF grid
(c) CoordX baseline
(d) CoordX grid
(e) Ground Truth
Figure 11: Images rendered using NeRF on Lego dataset.
21

---

Published as a conference paper at ICLR 2022
(a) NeRF bascline
(b) NeRF grid
(c) CoordX baseline
(d) CoordX grid
(c) Ground Truth
Figure 12: Images rendered using NeRF on Ship and Drums dataset.
22


Reviews:
[{"review_id": "UEPGR539W36", "reviewer": "Reviewer_xWSd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper are as follows:\n1. The quantitative results are communicated clearly, and the benefits of the method are well justified and ablated. \n    - The benefits in speed and lack of degradation in quality are shown across three signal fitting tasks (image, video, shape representation). It is also demonstrated that this idea can work for local implicit methods (using latent codes) and still provide a speed-up with limited degradation in quality. \n    - The degree of splitting versus number of layers comparison is insightful and shows that the method proposed is in fact more powerful than simply using smaller MLPs (i.e. the individual processing FC blocks for each coordinate are highly useful).\n2. The contribution of the paper is quite clear. I think that the simple idea of splitting coordinate processing, ablating it well and showing that it doesn’t degrade quality but speeds up the training and evaluation is quite clear. This makes it much more likely to be more widely adopted in practice, as faster training of coordinate-based representations of signals is a very important problem with lots of recent work.\n\nIn my opinion, the weaknesses of the paper are as follows:\n1. The exposition of the significance of the results, and the necessary contributions to obtain them. \n    - While fitting a single image, video, or 3D signal with a coordinate-based network is intellectually interesting as a toy problem, the main use of coordinate-based networks is in neural rendering. This means that it is essential for CoordX to be able to be applied to the volume rendering of NeRF-like methods, or the ray-tracing applied in neural surface based methods. However, to find any mention of volumetric methods using CoordX, I had to get to section C in the appendix. I think that this should absolutely be included and emphasized in the main paper, and ideally more experiments should be dedicated to this instead of only overfitting on the Blender lego scene. Additionally, this section should include more ablation on the properties of the modified volumetric rendering proposed. Without this, the magnitude of the contribution is severely limited, as simply being able to overfit a few types of signals faster isn’t that impactful to many applications.\n    - Similarly, section A of the appendix should be mentioned in a bit more detail in the main paper. The input point sampling is crucial for actually obtaining the speedup benefits of the CoordX architecture (just doing the same minibatch training as standard coordinate-based networks would not yield any speedup), and thus the contribution of CoordX should be both the modified architecture and the sampling method.\n2. Following weakness 1, first bullet point, I think that while the evaluation on the lego NeRF model is very crucial to include in the paper, I think that this should be studied more and more results should be demonstrated on more varied datasets. This would make the paper significantly stronger, because then CoordX architecture could be applied in many neural volume rendering systems, which have had a significant amount of work in recent years.\n\nMinor comments:\n- I think that the introduction could be reordered to set a better context for the contribution so as to not make it immediately confusing. For example, starting the first sentence with what CoordX does doesn’t make sense since I haven’t read about CoordX yet. Starting by introducing coordinate-based MLPs and their applications makes it more clear in what area the contribution of the paper will come from.\n- The notation is a bit confusing in equation (2). Shoudn’t $W_j$ also have an $i$ index, since this is an independent weight matrix for each branch?\n- It would be nice to have an ablation on the feature fusion method, as concatenation versus multiplicative modulation versus additive modulation has been a topic in generalizing over coordinate-based networks. This could certainly be interesting and enlightening on whether or not this is a crucial part of the CoordX architecture.\n", "summary_of_the_paper": "The paper proposes a novel coordinate-based network architecture which proposes to process each of the input coordinates independently in the first layer instead of together in a fully connected layer. This input style results in a speed-up in terms of evaluation of the network, and thus faster training and inference in tasks where coordinate-based MLPs are used, without incurring a significant degradation in terms of the quality of the signal fit. These benefits are demonstrated for the tasks of image representation, video representation, and 3D shape representation.", "summary_of_the_review": "Overall, the paper proposes an architecture modification which improves the evaluation time of coordinate-based MLPs without significantly hurting the quality of the signal fit. This contribution is demonstrated ablated clearly for signal memorization, and I believe that the proposed architecture has value for direct signal fitting tasks in coordinate-based networks. However, the effectiveness of the method is not immediately clear for applications in neural rendering, where coordinate-based networks are mainly used. A novel formulation of volume rendering which is compatible with the architecture change is proposed, and evaluated on one scene, which provides promising results that this can speed up the training of NeRF-like models in neural rendering, but this section of the appendix could be expanded upon significantly to create a significantly stronger paper. Based on the current state of the paper, I believe that the architecture shows promise and is ablated well for signal memorization, but may not be as significant as it could otherwise be. Thus, I believe it is marginally above the acceptance threshold.\n\n**Post Rebuttal Update**\n\nThe authors have done a good job of addressing my concerns in my original review. I think that while the paper does propose a small algorithmic change and relies on more complex sampling methods to work for general coordinate-based network applications (as noted by other reviewers), the results and contributions are ablated well and the simplicity of the idea could likely have a larger impact upon practical usage of coordinate-based networks. Thus, I have updated my score, and think that the paper in its current state should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OqpxuOQRu-", "reviewer": "Reviewer_MGAi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My take:\nWhen I first looked over this paper, I thought to myself, “That’s crazy! Does that actually work?” When I read it in detail, I thought, “Okay, the authors have definitely done their due diligence, and thought through a lot of the caveats needed to get this idea to work and to appropriately reap the benefits.” I then took to implementing it myself in a range of benchmark tasks involving implicit neural representations and in other architectures. As it turns out, this absolutely works as advertised, is extremely easy to implement, and provides the advertised compute speedups, albeit at a slight cost in parametric efficiency. What’s more, some digging turns up that there’s actually a reasonably solid theoretical justification for this approach: as noted by Hinton in arxiv.org/abs/2102.12627, “The Kolmogorov-Arnold superposition theorem states that every multivariate continuous function can be represented as a superposition of continuous functions of one variable.” I think this paper needs a bit of cleanup on the presentation and clarity side, but is clearly in accept territory. I am willing to champion this paper: while I don’t necessarily think its impact potential is massive, I think it does have the potential to have a reasonable degree of impact, and I especially believe that we should highlight solid work which teaches us something *surprising*.\n\n\nDetailed notes:\n- The authors mention that they require more parameters to reach a given performance level a few times, but I think they should be more upfront in specifying that this is a drawback of the method. I have experimented extensively with this and it is quite clear that, if one keeps a fixed reconstruction accuracy target, CoordX models require slightly more parameters than a non-CoordX model to reach that target regardless of architecture; this can also be seen in lines 1 and 3 of Table 5. As most readers/practitioners likely won’t care about a reduction in parametric efficiency due to the improvements in compute efficiency, I don’t think this will make the paper seem any weaker (and is more honest anyway). Specifically I would like to see mentioned in the second bullet point under contributions that the parameter count is *comparable* but is almost necessarily *slightly larger.* This also retains the specific design of the weight-sharing layers as a visible contribution. \n\n\n-One “drawback” of this model is that it does not improve things in the unidimensional coordinate case. This is not actually a drawback, but rather a lack of a benefit, but that’s perfectly fine. It might be worth mentioning this as e.g. audio representation would not benefit here.\n\nMinor:\n-The very first sentence of the introduction appears to be some sort of typo or accidentally left-in mistake (“recently coordx…”). Please fix this.\n", "summary_of_the_paper": "Summary: This paper proposes a modification to INR models on multidimensional coordinate grids where a subset of the earlier layers operate on the decomposed coordinate grid. In this setup, the grid (which is assumed to be regularly sampled to permit this decomposition) is broken into its constituent components, (e.g. x and y instead of (x,y)), passed through a single linear layer unique to that component, then through a stack of shared layers, followed by an outer product to return to the joint (x,y) space, and at least one layer that operates on the joint space. This approach lightly reduces parametric efficiency but strongly improves compute efficiency (both in terms of FLOPS, memory usage, and actual observed runtime) for common implicit neural representation tasks, including fitting images, videos, shapes, and volumetric rendering via radiance fields.\n", "summary_of_the_review": "This paper presents a technique of general interest to the subfield of implicit neural representations, has solid empirical results, and is in my opinion quite surprising. The paper is clearly in accept territory.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JeK7O_npIM", "reviewer": "Reviewer_KUjc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have no major issue with the paper. The idea is clearly presented with nice qualitative and quantitate support. The matrix decomposition discussion in Section 3.3 is pretty interesting to read. The algorithmic novelty, however, is rather limited. It's incremental, but delivers good results.", "summary_of_the_paper": "The paper proposes an interesting tweak to the network architecture to accelerate CoortMLP. The idea is to split the input coordinates along the dimensions and then share weights before fusion.  The authors analyze the theoretical upper bound (as far as the MAC ops are concerned) and show about 2X speedup on actual machines.", "summary_of_the_review": "Novelty is limited. Practical engineering efforts with wide applicability and good results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bb-r5fXy2F9", "reviewer": "Reviewer_z7hu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I think the motivation of this paper is interesting and I appreciate the many experiments the authors have run, I still believe there are quite a few issues with the paper which need to be addressed. The main strengths and weaknesses (in my eyes) are described below.\n\n**Strengths**:\n- The motivation of the paper is good and improving the speed of implicit reps is an important topic. The observation that building methods that leverage the locality between coordinate points could help implicit reps (as opposed to regular implicit reps that consider points as independent elements of a batch) is interesting\n- The authors perform experiments on a wide variety of data modalities (images, videos, 3D shapes, NeRF scenes) and make some interesting ablations\n\n**Weaknesses**:\n- In general, the experimental results are not very convincing. For example, on the image experiments, the training time can be halved but at the cost of a 3dB drop in PSNR. The authors refer to this as “a slight drop in quality” but 3dB is a very significant drop (PSNR is measured on a log scale). For videos and the superresolution experiments the reconstruction results are on par with the baselines but are faster, which is good. However, comparisons to e.g. [1] or other methods aiming to accelerate implicit reps are missing. Methods such as these would very likely both be faster and have better reconstruction accuracy than the proposed method. Adding ablations against more than just vanilla implicit reps models would definitely help strengthen the paper.\n- One of the main drawbacks of the proposed method is that the decomposition operation requires a regular grid. A huge part of the motivation for using implicit representations is exactly to be able to model data that does not lie on a regular grid, e.g NeRF scenes, meshes and so on. So this does seem to be a significant drawback. While the authors briefly discuss this and propose a fix for NeRF scenes, this fix still relies on the existence of some underlying grid (256^3) which then limits the resolution of the signal.\n- The related work section disregards some very closely related work in accelerating training + inference of implicit representations [1]. While the authors cite this paper among others, they then mention that competing methods either are only applicable to a single task (not the case for [1]), have high memory overheads (not the case with [1], in fact memory is reduced) and they are not effective for dense/rich signals like images (not the case for [1], which can model gigapixel images). While [1] can arguably be considered concurrent work, I believe there should be a more fair discussion of this.\n- The paper writing is not very clear. The authors introduce a lot of notation which can make the paper heavy to read. In addition, the authors seem to use very non-standard notation. For example, in equation (1) and (2) it appears like the authors use ; to denote matrix multiplication which I’ve never seen before. In equation (3) the outer product is denoted with a typical product symbol, it would be much clearer to just write this as an outer product.\n- The paper seems rushed and there are many typos. For example, it looks like the first sentence of the paper is misplaced.\n- Comparisons on e.g. NeRF scenes do not seem fair as there is a lot of work accelerating the rendering of NeRF scenes which considerably outperforms the proposed method (including the papers [2, 3] that the authors already cite). It would be nice to add comparisons to these too.\n\nThe authors also mention in the conclusion that decomposing signals into smaller signals that can be used to reconstruct the original signal could potentially be applied to data compression. This was one of the first thoughts I had when initially reading the paper and I think it would be interesting to expand more on this, e.g. in the context of [4].\n\n[1] ACORN: Adaptive Coordinate Networks for Neural Scene Representation, Martel et al.\n\n[2] FastNeRF: High-Fidelity Neural Rendering at 200FPS, Garbin et al.\n\n[3] PlenOctrees for Real-time Rendering of Neural Radiance Fields, Yu et al.\n\n[4] COIN: COmpression with Implicit Neural representations, Dupont et al.\n", "summary_of_the_paper": "This paper proposes a new architecture for implicit neural representations, called CoordX, which splits each dimension of the input signal into separate branches (e.g. the x and y coordinates of pixel locations in an image) and processes each of these separately before fusing them. The authors achieve this by projecting each of these branches into a hidden feature and then using shared fully connected layers to process these. Each branch is then fused by an outer product which then reconstructs the entire input grid (e.g. for an image of size H x W, 2 branches take in H locations and W locations respectively which are then combined into H x W features by the fusion operation). The fused layers are processed by a few more MLP layers to output the predicted features. In addition, the authors propose a method for effectively subsampling the grid during training as well as different splitting strategies for the branches.\n\nThe authors perform experiments on various data modalities, including images, videos, 3D shapes and NerF scenes.\n\nThe main contributions of the paper in my eyes are then:\n- Introducing a new architecture for implicit neural representations that in certain cases can improve training/inference speed without incurring a decrease in reconstruction\n- Experiments on various data modalities showing the strengths/weaknesses of the method\n", "summary_of_the_review": "This paper introduces a new architecture for implicit neural representations. While there are some interesting ideas in the paper, overall I still believe more work needs to be done and better comparisons to baselines need to be made before this paper is ready for publication.\n\n--- Post rebuttal update ---\n\nI appreciate the large efforts the authors have put in to address the issues put forward in this review and by the other reviewers. With these changes I believe the paper is stronger and have updated my score accordingly. In particular I appreciate the experiments showing that CoordX can also help approaches such as ACORN (even though these results still seem preliminary, particularly for Tokyo). Including these results in the updated paper will definitely help give a clearer picture of the contributions. The extended discussion and experiments around NeRF scenes are also helpful and appreciated. \n\nWhile I still believe there are some drawbacks with the proposed approach (in particular that using the method for non grid-based signals requires slightly hacky workarounds and that the gains by the proposed method are quite small compared to other methods aiming to accelerate implict reps e.g. ACORN), I am happy for this paper to be accepted in its current updated state.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Implicit neural representations are a new and promising method to represent images and scenes. Implicit neural representations enable good performance on task like view synthesis. Those networks generate an image of scene pixel-by-pixel and are therefore computationally expensive. The paper proposes a method to accelerate inference with an MLP by learning each dimension of the input (e.g., x, y, and z coordinates) separately. The paper reports speedups by a factor of up to three.

The four reviewers all agree that this paper should be accepted. The reviewer's highlight that the speedup is a solid technical contribution, and agree that the idea of splitting the coordinates is well motivated and well evaluated, and leads to the advertised speedups. 
During the review process, the reviewers also raised some issues, such as a slight performance loss at that cost of a slight increase in efficiency, but were convinced by the response of the reviewers. 

I recommend accepting the paper. Like the reviewers, I think that the proposed idea is interesting and technically sound, however, I'm a bit concerned about the slight drop in performance: If a slight drop of performance is allowed, then a speedup is possible by simply making the networks smaller, reducing the layers, or through other more immediate means than the proposed one. The contribution would be even more convincing if the paper also compares the speedup in a fair setup where the performance is kept constant.
Decision: Accept (Poster)