ResFields: Residual Neural Fields for Spatiotemporal Signals
Authors: Marko Mihajlovic
Sergey Prokudin
Marc Pollefeys
Siyu Tang
Venue: ICLR 2024 spotlight (2024)
Keywords: neural fields
NeRF
reconstruction
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A novel time-dependent layer for MLPs to improve capturing and reconstruction of spatiotemporal signals.

Abstract: Neural fields, a category of neural networks trained to represent high-frequency signals, have gained significant attention in recent years due to their impressive performance in modeling complex 3D data, such as signed distance (SDFs) or radiance fields (NeRFs), via a single multi-layer perceptron (MLP). However, despite the power and simplicity of representing signals with an MLP, these methods still face challenges when modeling large and complex temporal signals due to the limited capacity of MLPs. In this paper, we propose an effective approach to address this limitation by incorporating temporal residual layers into neural fields, dubbed ResFields. It is a novel class of networks specifically designed to effectively represent complex temporal signals. We conduct a comprehensive analysis of the properties of ResFields and propose a matrix factorization technique to reduce the number of trainable parameters and enhance generalization capabilities. Importantly, our formulation seamlessly integrates with existing MLP-based neural fields and consistently improves results across various challenging tasks: 2D video approximation, dynamic shape modeling via temporal SDFs, and dynamic NeRF reconstruction. Lastly, we demonstrate the practical utility of ResFields by showcasing its effectiveness in capturing dynamic 3D scenes from sparse RGBD cameras of a lightweight capture system.

Published as a conference paper at ICLR 2024
RESFIELDS: RESIDUAL NEURAL FIELDS FOR SPATIOTEMPORAL SIGNALS
Marko Mihajlovic', Sergey Prokudin Mare P Siyu Tang'
ETH Zurich'; Microsoft²; ROCS, University Hospital Balgrist, University of Zürich
ma markomih. stxkomii.qithub...
ABSTRACT
Neural fields, category of neural networks trained to represent high-frequency signals, have gained significant attention recent years due to their impressive performance in modeling complex 3D data, such signed distance (SDFs) radiance fields (NeRFs), via a single multi-layer perceptron (MLP). However, de- spite the power and simplicity of representing signals with MLP, these methods still face challenges when modeling large and complex temporal signals due to the limited capacity of MLPs. this paper, we propose an effective approach ad- dress this limitation by incorporating temporal residual layers into neural fields, novel class of networks specifically designed to effec- tively represent complex temporal signals. We conduct comprehensive analysis of the properties of ResFields and propose matrix factorization technique to re- duce the number of trainable parameters and enhance generalization capabilities. Importantly, our formulation seamlessly integrates with existing MLP-based neu- ral fields and consistently improves results across various challenging tasks: 2D video approximation, dynamic shape modeling via temporal SDFs, and dynamic NeRF reconstruction. Lastly, we demonstrate the practical utility of ResFields by showcasing its effectiveness capturing dynamic 3D scenes from sparse RGBD cameras of lightweight capture system.
dubbed ResFields.
INTRODUCTION
Multi-layer Perceptron (MLP) is common neural network architecture used for representing con- tinuous spatiotemporal signals, known as neural fields. Its popularity stems from its capacity to encode continuous signals across arbitrary dimensions (Kim & Adali 2003). Additionally, inher- ent implicit regularization (Goodfellow et al. 2016 Neyshabur et al. 2014) and spectral bias (Ra-) haman et al.] 2019) equip ML.Ps with excellent interpolation capabilities. Due to these remarkable properties, MLPs have achieved widespread success in many applications such as image synthesis, animation, texture generation, and novel view synthesis (Tewari al. 2022 Xie et al., 2022).
However, the spectral bias of MLPs (Rahaman et al. 2019), which refers to the tendency of neural networks learn functions with low frequencies, presents challenge when it comes to accurately representing complex real-world signals and capturing ine-grained details. Previous efforts have aimed address the spectral bias by utilizing techniques like positional encoding (Vaswani al.) 2017 Mildenhal et al. 2020 Zhong et al. 2019 Müller et al., (022) or special activation functions (Sitzmann al.. 2020b Fathony et al. 2020). However, even with these methods, representing fine- grained details remains challenge, particularly when dealing with large spatiotemporal signals such as long videos or dynamic 3D scenes.
A straightforward way of increasing the capacity of MLPs is to increase the network complexity in terms of the total number of neurons. However, such an approach would make the inference and optimization slower and more GPU memory expensive, time and memory complexity scales with respect to the total number of parameters. Another possibility is to meta-learn MLP weights Sitzmann et al. 2020a) and maintain specialized independent parameters, but this imposes slow training that does not scale to photo-realistic reconstructions (Tancik et al.) (021). By far the most popular approach for increasing modeling capacity is partition the spatiotemporal field and fit "Code, data, and pre-trained models relcased

---

Published as a conference paper at ICLR 2024
Siren Sitzmann et al. 2020b
TNeRF Li ct al. 2022
Baseline
+ResFields
(a)
(b)
(c)
Figure 1: ResField extends an MLP architecture effectively represent complex temporal signals by replacing the conventional linear layers with Residual Field Layers. As such, ResField is versatile and straightforwardly compatible with most existing temporal neural fields. Here we demonstrate its applicability on three challenging tasks by extending Siren (Sitzmann et al. (222b) and TNeRF (Li (ct al. 2022): (a) learning temporal signed distance fields and (b) neural radiance fields from four RGB views and (c) from three time-synchronized RGBD views captured by our lightweight rig. The figure is best viewed in electronic format on a color screen, please zoom-in to observe details.
separate/local neural fields (Reiser et al., 2021 Müller et al. 2022 Chen et al.) 2022 these approaches hinder global reasoning and generalization due local gradient updates of grid structures (Peng et al. 2023).
However,
The challenge that aim to address is how to increase the model capacity in way that agnostic the design choices of MLP neu- ral fields. This includes architecture, input en- coding, and activation functions. At the same time, we must maintain the implicit regular- ization property of neural networks and retain compatibility with existing techniques devel- oped for reducing the spectral bias (Mildenhall
ResField Layer
ResField Layer
ResField Layer
(t.x)
y
Wi
Wi
Wi
Wi(b
Wi(t)
Wa(t)
Figure 2: ResField MLP Architecture.
al.) 2020 Sitzmann et al. (2020b). Our key idea is substitute MLP layers with time-dependent
layers (see Fig. 2 whose weights are modeled as trainable residual parameters W.(() added to the existing layer weights W, We dub neural fields implemented in this way ResFields.
Increasing the model capacity in this way offers three key advantages. First, the underlying MLP does increase in width and hence, maintains the inference and training speed. This property is crucial for most practical downstream applications of neural fields, including NeRF (Mildenhail al. 2020) which aims solve inverse volume rendering (Drebin et al. 1988) by querying neural fields billions of times. Second, this modeling retains the implicit regularization and generalization properties of MLPs, unlike other strategies focused on spatial partitioning (Reiser al. [202] Müller
et al. 2022 Peng et al. 2023 Işik et al. 2023). Finally, ResFields are versatile, easily extendable, and compatible with most MI. bhased methods for spatiotemporal signals.
However, the straightforward implementation of ResFields could lead to reduced interpolation prop- erties due to a large number unconstrained trainable parameters. this end, inspired by well- explored low-rank factorized layers (Denil et al. 2013 Toannou et al. 2015 Khodak et al.) [021], we propose to implement the residual parameters as global low-rank spanning set and a set of time- dependent coefficients. As we show in the following sections, this modeling enhances the gen- eralization properties and further reduces the memory footprint caused by maintaining additional network parameters.
2

---

Published as a conference paper at ICLR 2024
In summary, our key contributions are:
We propose an architecture-agnosti building block for modeling spatiotemporal fields that we dub ResFields.
We systematically demonstrate that our method benefits a number of existing methods: Sitzmann
et al. (2020b); Pumarola et al. (202T); Park et al. (2021a b): Li et al. (2022); Cai et al. (2022) Cao
& Johnson (2023); Fridovich- Keil et al. (202))
We validate ResFields on four challenging tasks and demonstrate state-of-the-art (Fig. D 2D
video approximation, temporal 3D shape modeling via signed distance functions, radiance field reconstruction of dynamic scenes from sparse RGB(D) images, and learning 3D scene flow.
RELATED WORK
Neural field is field physical quantity that has value for every point in time and space- that is parameterized fully or in part by neural network (Xie et al. 2022). typically an MLP as the universal approximator (Kim & Adali 2003). However, straightforward fitting of signals to regular MLPs yields poor reconstruction quality due to the spectral bias of learning low frequencies (Rahaman et al., 2019). Even though this issue has been alleviated through special imput encodings [Mildenhall et al.) 22200 Barron et al., [22TT] 2022) or activation functions (Sitzmann et al. 2020b]
Tancik et al. 2020; Fathony et al., 2020 Lindell et al., 2022 Shekarforoush al.] 2022). neural fields still cannot scale to long and complex temporal signals due to the limited capacity. A natural
way of increasing the modeling capacity is to increase the network's size in terms of the number of
parameters. However, this trivial solution does not scale with GPU and training time requirements.
Hybrid neural fields leverage explicit grid-based data structures with learnable feature vectors to improve the modeling capacity via spatial (Takikawa et al.) [021] Müller et al. [022] Chen et al.
2022
Chan et al.) 2022) and temporal (Shao et al.|| [023] Fridovich- Meelovicch--elleee et al. 2023 Cao & Johnson
2023 Peng al. 2023) partitioning techmiques. However, these approaches sacrifice the desired
global reasoning and implicit regularization (Neyshabur et al.) 2014; Goodfellow et al.) 2016) that needed for generalization, especially for solving ill-posed problems like inverse rendering. In contrast, our solution, ResFields, focuses on improving pure neural network-based approaches that still hold state-of-the-art results across several important applications, as we will demonstrate later.
Input-dependent MLP weights is another common strategy for increasing the capacity of MLPs by directly regressing MLP weights, c.g. via a hypemnetorr (Mehta et al.) [021 Wang al. 2021c) or a convolutional (Peng et al.] 2023) neural network. However, these approaches introduce an ad- ditional, much larger network that imposes a significant computational burden for optimizing neural fields. KiloNeRF (Reiser et al. 2021) proposes to speed up the inference of static neural radiance fields by distilling the learned radiance field into a grid of small independent MLPs. However, since a bigger MLP is still used during the first stage of the training, this model has the same scaling limi- tations as the original NeRF, Closest in spirit to our approach, the level-of-experts (LoE) model (Hao) et al. 2022) introduces an input-dependee hierarchical composition of shared MLP weights the expense of reduced representational capacity. Compared to LoE, ResFields demonstrate stronger generalization and higher representational power for modeling complex spatiotemporal signals.
Temporal fields are typically modeled by feeding the time-space coordinate pairs to neural fields. SIREN (Sizzmann et al. 2020b) was one of the first neural methods faithfully reconstruct 2D video signal. However, scaling this approach to 4D is infeasible and does not produce desired results as demonstrated in dynamic extensions of NeRF models (Pumarola et al. 2021 Li et al. 2022). Therefore, most of the existing NeRF solutions (Pumarola et al.) 2021 Park et al. 202 la decouple the learning problem into learning static canonical neural field and deformation neural network that transforms a query point from the observation to the canonical space where the field queried. However, these methods tend to fail for more complex signals due to the difficulty of learning complex deformations via a neural network, as observed by [Gao et al. (2022). To alleviate the problem. HyperNeRF (Park et al. 202 introduced an additional small MLP and per-frame learnable ambient codes to better capture topological variations, increase the modeling capacity, and simplify the learning of complex deformation. The recent NDR (Cai et al. 2022). a follow up work of HyperNeRF, further improves the deformation field by leveraging invertible neural networks and more constrained SDF- density formulation (Yariy et al. 2021). All of these methods are fully compatible with the introduced ResFields paradigm which consistently improves baseline results.
3

---

Published as a conference paper at ICLR 2024
Scene flow is commonly used model the dynamics of neural fields. Most works use MLPs to
model scene flow by predicting offset vectors (Pumarola et al. 2022 ei al. 202 Prokudin et al.
2023
Wang et al. (023b), SE(3) transformation (Park al. [202 la Wang et al. [2023a), coefficients of pre-defined bases Wang et al. 2021a Li et al., 2023). or directly using invertible architectures
(Cai et al., 2022 Wang et al. 2023c). These representations are compatible with ResFields which
further enhance their learning capabilities.
Residual connections have long history in machine learning. They first appeared in Rosenblat (196T) in the context of coupled perceptron networks. Rosenblatt's insight was that the residual connections increase the efficiency of responding to imput signals. Since then, residual connections have been extensively studied and found a major practical utility as solution to training deep neural networks by overcoming the vanishing gradient problem (Hochreiter 1998, Srivastava al.) 2015 He et al. 2016) and became a de facto standard for modeling neural networks. Unlike these residual connections that are added to the output of MLP layers, our ResField layers model the residuals of the MLP weights, which in turn yields higher representation power of neural fields, making them more suitable for modeling complex real-world spatiotemporal signals. To the best of knowledge, directly optimizing residual or multiplicative conrectives of model parameters has been explored in the context of fine-tuning large language models (Karimi Mahabadi et al.) [202]] Hu et al. 2021 Dettmers al. 2023) or predicting model weights (Wang et al. [2021c), and has not been explored for directly training spatiotemporal neural fields.
3 RESFIELDS: RESIDUAL NEURAL FIELDS FOR SPATIOTEMPORAL SIGNALS
Formulation. Temporal neural fields encode continuous signals f Rd x R + R via neural network Lg. where the input is a time-space coordinate pair E R, x R2) and the output is field quantity y E R. More formally, the temporal neural field is defined as:
Doll, x) = on (Wa o Φn- ................................ o 61) (1, x) + bn),
(1, oi (W. x. + b;),
where o, RRN + RM is the ith layer of the MLP, which consists of the linear transformation by the weight matrix W; E x,M, and the bias b, E R.N. applied to the input E IR M, followed by non-linear activation function og, The network parameters 0 are optimized by minimizing a loss term L directly w.r.t ground truth signal or indirectly by relating field quantity to the sensory input, e.g. via volume rendering equation for radiance field reconstruction.
Limitations of MLPs. To model complex and long signals, it is crucial for the underlying MLP to have a sufficient modeling capacity, which scales with the total number parameters. However, as the MLP size increases, the training time of neural fields becomes slower while increasing the GPU memory requirements, ultimately leading to the bottleneck being the MLP's size. This especially highlighted for dynamic radiance field reconstruction which requires solving inverse rendering problem through billions of MLP queries. In the following, we introduce ResFields, an approach for alleviating the capacity bottleneck for modeling and reconstructing spatiotemporal signals.
ResFields model. We introduce residual field layers (Fig, 2 to effectively capture large and complex spatiotemporal signals. ResFields, MLP that uses at least residual field layer, alleviates the aforementioned capacity bottleneck without increasing the size of MLPs in terms of the number of layers and neurons. In particular, we replace linear layer of an MLP &i with temporal iime-conditioned residual layer defined as:
(t xx) = a(W(W + W,(I))x; + +b,,,
(3)
where Wi(1)= R + N.M,M is time-dependent and models residuals of the network weights. This simple formulation increases the model capacity via additional trainable parameters without modifying the overall network architecture.
Mi
R
M
ResFields factorization. However, naively imple- menting (ttt) E N X,,, as a dictionary of train- able weights would yield vast amount of indepen- dent and unconstrained parameters. This would result in partitioning of spatiotemporal signal, akin to the
T
V
X
M;
T W;
Figure 3: Factorization of W..
4

---

Published as a conference paper at ICLR 2024
Sires- ResPields
Siren-256
Training Ieeraiios
Training Time.:
25 min
52 Min
Text PSNR
33,
27.3
Figure 4: 2D video approximation. Comparison of diffferen neural fields fitting RGB videos. The training and test PSNR curves (left and right respectively) indicate the trade-off between the model's capacity and generalization properties. Instant NGP offers good overfitting capabilities, however, it struggles to generalize to unseen pixels. A Siren MLP with 1024 neurons (Siren-10244 shows good generalization properties, however, lacks representation power (low training and low test PSNR). A smaller Siren with 512 neurons implemented with ResFields (arre-552+Reesiebhe demonstrates good generalization while offering higher model capacity. Besides the higher accu- racy, our approach offers approximately 2.5 times faster convergence and 30% lower GPU memory requirements due to using smaller MI.P (Tab. D.. Results on the right provide visual comparison of Siren with 256 neurons and Siren with 128 neurons implemented with ResField layers.
space partitioning methods (Reiser et al. 2021; Müller et al. 2022 Shao et al. 2023), and hinder a
global reasoning and implicit bias of MLPs, essential properties for solving under constrained prob- lems such as a novel view synthesis from sparse setups. To this end, inspired by well-established low-rank factorized layers (Denil et al. [013 loannou et al. 2015 Khodak al. (021), we directly optimize time-dependent coefficients and R, RR-dimensional spanning set of residual network weights that are shared across the entire spatiotemporal signal (see Fig. 3 In particular, the residual of network weights are defined as
Wi(t) - & v((r)() -M,
where the coefficients vtt) E R. and the spanning set M E RR are trainable parameters; square brackets denote element selection. To model continuous coefficients over the time dimension, rows. Such formulation reduces
we implement V E x&, as a matrix and linearly interpolate the total number of trainable parameters and further prevents potential undesired overfitting that is common for field partition methods as we will demonstrate later (Sec. 4.5).
Key idea. The goal of our parametrization is to achieve high learning capacity while retaining good generalization properties. To achieve this for a limited budget in terms of the number of parameters, allocate as few independent parameters per time step (in v(t) and many globally shared parameters M. Allocating more capacity towards the shared weights will enable /) the increased capacity of the model due to its ability discover small patterms that could be effectively compressed in shared weights and 2) stronger generalization as the model is aware of the whole sequence through the shared weights. Given these two objectives, design v()) have very few parameters (Rt,) and M to have most parameters (ixxNxxM,,) see the Sup. Mat. for further implementation details.
4 EXPERIMENTS
To highlight the versatility of ResFields, we analyze our method on four challenging tasks: 2D video approximation via neural fields, learning of temporal signed distance functions, radiance reconstruc- tion of dynamic scenes from calibrated RGB(D) cameras, and learning 3D scene flow.
4.1
2D VIDEO APPROXIMATION
Learning mapping of pixel coordinates to the corresponding RGB colors is popular benchmark for evaluating the model capacity of fitting complex signals (Müller et al.) 2022 Sitzmann et al.) 2020b). For comparison, consist respectively of 250 and 300 frames (with resolutions at 512 x 512 and 272 x 640) and fit neural representations by minimizing the mean squared error ground truth RGB values.
two videos (bikes and cat from Sitzmann et al. (2020b)) that
5

---

Published as a conference paper at ICLR 2024
Unlike the proposed setup in Sitzmann et al. (2020b) where the focus is pure overfitting to the image values, our goal is to also evaluate the interpolation aspects of the models. For this, we leave out 10% of randomly sampled pixels for validation and fit the video signal on the remaining ones. We compare our approach against Instant NGP, a popular grid-based approach to neural field modeling, with the best hyperparameter configuration for the task (see supplementary). We also compare against a five-layer Siren network with 1024 neurons (denoted as Siren-1024), as pure MLP-based approach. For our model, we choose a five-layer Siren network with 512 neurons, whose hidden layers are implemented as residual field layers with the rank R; = 10 for all hidden layers (Siren- 51 12+ResFields). We refer to the supplementary for more details and ablation studies on the number of factors, ranks, and layers for the experiment.
Table I: Video approximation.
Insights. We report training and test PSNR values averaged over the two videos in Fig Tab. Here, Instant-NGP of- fers extremely fast and good overfitting abilities as storing the data in the hash grid structure effectively resolves the problem of limited MLP capacity, alleviating the need for residual weights. This, however, comes at the ex- pense of the decreased generalization capability. ftren-1024 has good generalization properties, but clearly underfits the signal and suffers from blur artifacts. Unlike Siren-1024, Siren-512 with Res- Fields offers significantly higher reconstruction and generalization quality (36.37 vs 39.21 PSNR) while requiring 30% less GPU memory and being about 2.5 times faster to train.
test PSNR1 4[w/k]t GPU)
NGP
34.52
131
166
Siren-1024 Sirem-51--Resiclld
36.37
3.55
9.7G
39.21
9.78
6.5G
This simple experiment serves proof of concept and highlights our ability fit complex tem- poral signals with smaller MLP architectures, which has a significant impact on the practical down- stream applications we discuss in the following sections.
4.2 TEMPORAL SIGNED DISTANCE FUNCTIONS (SDF)
higned-distance functions model the orthogonal distance of a given spatial coordinate x the sur- face of a shape, where the sign indicates whether the point is inside the shape. We model temporal sequence of signed distance functions via a neural field network that maps time-space coordinate pair (t E R. x E IR3) to a signed distance value (y E R).
We sample five sequences of different levels of difficulty (four from Deforming Things (Li et al., (2211)) and one from ReSynth (Ma et al. 2021) and convert the ground-truth meshes to SDF values. We supervise all methods by the MAPE loss following Müller et al. (2022). To benchmark the methods, we extract a sequence of meshes from the learned neural fields via march- ing cubes (Lorensen & Cline 1987) and report LI Chamfer dis- tance (CD.) and normal consistency (NCL) w.r.t the ground-truth meshes (scaled by 102 and 102 respectively). As a main baseline, use the current state-of-the-art Siren network (five layers) and compare it against Siren implemented with our ResField layers. where residual field layers are applied to three middle layers. We empirically observe that using ResField on the first and last layers has a marginal impact on the performance since weight matrices are small and do not impose a bottleneck for modeling capacity.
Table 2: Temporal SDF
Rank Resouvcss
R: GPU [lusl]
CDL NDJ
Siren-128
2.4 20.06
15.06
5
9.47 18.54
8.79 16.61
+RosFiclds
2.5G 20.25
20
8.43 15.48
40
8.16 14.19
Siren-256
356
16.37
5
7.50 13.00
+ResFields
20
10
3.8G
48.19
7.66 11.84
7.71 12:24
40
7.67 11.67
Insights Quantitative and qualitative results (Tab. 2 Fig. [ demonstrate that ResFields consistently improve the reconstruction quality, with the higher rank increasingly improving results. Importantly, we observe that Siren with 128 neurons and ResFields (rank 40), performs better compared to the vanilla Siren with 256 neurons, making our method over two times faster while requiring less GPU memory due to using much smaller MLP architecture. Alleviating this bottleneck is of utmost importance for the reconstruction tasks that require solving inverse rendering by querying the neural field billions of times as we demonstrate in the next experiment.
4.3 TEMPORAL NEURAL RADIANCE FIELDS (NERF)
Temporal or Dynamic NeRF represents geometry and texture neural field that models function of color and density. The model is trained by minimizing the pixel-wise error metric between the images captured from known camera poses and ones rendered via the differentiable ray marcher
6

---

Published as a conference paper at ICLR 2024
Table 3:
Temporal radiance field reconstruction Owlii (Xu et al. 2017). Previous state-of- the-art methods consistently benefit from ResField layers without imposing high computational overhead; bold numbers denote best per-sequence performance while colors denote the overall 1st.
2nd and 3rd best-performing model; denotes which layers are substituted with ResFeilds layers.
Meon
Basketball
Model
Daneer
Excrcise
HPST
CDJ
SSIM
CDJ SSIM PSNRT
SSIM PSNKT
CDJ
SSIM PSNK? CDJ SSIM PSNKt
Nous2 (Way al. 2023
0.5h
30
69.4
91.01
21.73
75.7
90.57 20.48
65.6
39.64
23.01
77.1
93.16
23.23
59.2
90.67
20.21
TensoriD
p00d
15h
0.035
32.8
91.05
22.59
30.5
91.22
22.51
40.3
89.30
22.46
26.7
91.53
23.24
33.5
92.16
22.16
HexxPlaee Coo & Johnson 2023
0.359
20.9
92.62 24.71
17.3 93.22 25.13
24.2
91.48
25.23
23.5 91.88
23.53
18.8
93.92
24.96
ResFicids
93.51
19.7
94.30
25.33
ResFields (6
2,
0.354
17.6 93.74 25.79
14.5 94.47 26.62
21.7 92.61
25.82
18,9 93,47 25.14
15.7
94,69
25.82
NyNeRR Ti
2022
0.328
31.0 91.95 23.59
28.0 92.56
23.49
44,9 89.84
23.11
30.7
91.54
23.33
20.3
93.88
24.45
ResFields
0.327
20.8 93.69 25:57
14.7 94.58
26.54
26, 92.24
25.36
24.6
93.35
25.20
17.7
94.59
25.17
ResFields (8 11:11,22
2,
0.323
19.3 93.81 25.40
20.3 93.49 24.77
22.2 93:07
26.16
17.6
93.69
25.22
17.1 94,99
25.80
ResFields (2 1.....77
0.316
94.00
93.15
25.28
25.13
TNeRI' cr
0.339
17.2 94,18 26.18
15.1 94.57 26.33
20.2 93.31
26.52
19.3 93.53 25.09
14.1 95.33
26.77
+ ResFields
12h
0.339
14.6 94.99 27.15
12.1 95.67 27.98
18.5 94.07
27.23
14.9 94.59
26.20
13.0
95.63
27.19
ResFields (i
0.334
14.2 95.21
27.44
12.2 95.84
27.98
18.3 94.33
27.81
13.3 94.87 26.55
27.40
+ -ReeFieldds (8 1.....77
0.328
14.2 95.45
27.55
12.2 95.90
27.82
17.8 94.49
27.45
14.5
95.22
26.82
12.3 96.21
28.11
DNcKF (Pumomila
2021
0.215
92.09
23.36
22.3 93.21
24.74
90.51
23.19
38.5
91.17
93.47
24.21
Restields
0.214
14.2 16,16 27.33
12.1 95.88
28.26
18.1 95
27.03
14.1 94.66 26.24
12.8 95.95
27.79
ResFields (i
1,233)
0.213
14.0 95.34 27.60
12.2 95.95 28.
17.6 94.45
27.84
14.0 94.88 26.40
12.4
96.08
27.97
RRsFFellds
0.210
28.34
17.3 94.85
27.83
14.3 95.45 27.25
28.14
Nerfies (Park al
2021
0.180
23.2 93.15 24.35
21.1 93.53 24.74
28.2 92.02
24.25
23.8 92.96 23.81
19.7 94.09
24.60
+ ResFFelds
24h
0.180
14.6 92.12 27.26
12.3 95.64
86
19.3 93.95
26.91
14.2 95.10 27.00
12.7
95.77
27.29
ResFields [i
0.179
14.0 95.32 27.43
11.9 95.78
27.87
18.6 94.30
27:21
13.0 95.27 27.11
12.5 95.91
27.51
+ ResFields [0
......7)
0.177
13.8 95.57
27.72
11.8 95.79 27.42
17.6 94.68
27.78
13.5 95.67 27.73
12.2 96.16
27.94
HyperNeRF
2021N
27.15
ResFields [
0,144
14,4 98.18 27.36
12.4 95.73 28.05
18.7 94, 18 27.38
13.4 95.15 26.96
13.0 95.65
27.05
32h
+ ResFiclds (1
0.144
95.35
27.45
12.4 95.86
28.11
18.4 94.36
27.32
12,9
95.35
27.24
12.8
95.82
27.14
ResFields
0.143
14.2
95,50
27.64 64
11.9 95.77 27.54
18.0 94.63
27.76
14.3
95.38
27.11
12.4
96.24
28.16
NDR Cai etal
2022
0.129
15.3 94.82
26.78
13.2
95.36 21.31
19.7 93.98 26.95
15.4
94.27
23.69
12.9
95.65
27.18
95.14
18.2
27.17
34h
ResFields (1
2,3)
0.129
14,0
95.36
27.
12.2
96.00
28.31
18.1 94.29
27.21
13.2
95.12
26.87
12.6
96.04
27.81
+ ResFields (1= 1.....7)
0.127
14.2
95.56
27.81
11.9 96.02
28.02
19.3 94.48
27.51
13.1 95.38
27.11
12.4
96.36
28.61
DyNeRF
DNeRF
Norfies
TNdRF
HyperNeRF
NDR
GT
Pummmola SNIK
2021
202 CD
2022
Park
202Th
SSIMETSNI
CDL
EDT
SSAUPSNK
SSIMPSNKI
CDL
SSENIPSKK
CD
SSINEFSSR
Base
92.56/23449
93.53/24.74
94.57/26.33
95,47/27.44
13,0
95,36/2..11
ResFiclds
94,47/25..3
17.9
96.15/28.34
12.9
95.79/7.42
11.8
95.90/27.82
12.2
95.77/27.54
11.9
96.02/28.02
11.9
Baseline
91,54/23.33
30.7
92.56/23:49
28.0
92 96.6613.81
23,5
$1.57/6.33
15.1
95,47/27.41
13.0
95.36/27.31
13.2
RestFieldk
91,54/23.33
96.45/27.25
95.67/27.73
95.22/26.82
05.38/27.11
14.3
95.38/27.11
Figure 5:
Temporal radiance fields on Owlii (Tab. 3)
metrics are averaged across all test views.

---

Published as a conference paper at ICLR 2024
(Mildenhall et al., 2020 To better model geometry, adopt the MLP architecture and signed distance field formulation from VolSDF (Yary et al. (021) that defines density function as Laplace's 's cumulative distribution function applied to SDF. We refer readers to the supplementary for the results with the NeRF backbone and further implementation details.
Following Waan et al. 2021b), all models are supervised by minimizing the pixel-wise difference between the rendered and ground truth colors (4 error), the rendered opacity and the gt mask (binary cross-entropy), and further adopting the Eikonal (Gropp et al. 2020)) mean squared error loss for well-behaved surface reconstruction under the sparse capture setup:
L = Leolor + A Lig + A2hmark-
We use four sequences from the Owlii (Xu et al.) 2017) dataset to evaluate the methods. Compared to fully synthetic sequences previously utilized for the task (Pumarola et al., 202T). the dynamic Owlii sequences exhibit more rapid and complex high-frequency motions, making it harder task for MLP-based methods. At the same time, the presence of ground truth 3D scans allows us to evaluate both geometry and appearance reconstruction quality, as compared the sequences with only RGB data available (Li et al., 20222 Shao et al., 2023). We render 400 RGB training images from four static camera views from 100 frames/time intervals and 100 test images from a rotating camera from 100 frames. We report LI Chamfer distance (CD.) (scaled by 10") and the standard image-based metrics (PSNR1, SSIM
We benchmark recent state-of-the-art methods and their variations implemented with ResField lay- ers of rank ten (R, 10) TNeRF (Pumarola et al.] 202T: Li et al.) 2022). DyNeRF (Li et al.
2022). DNcRF (Pumarola al. 2021). Nerfies (Park et al.]| 202 202la), la). HyperNeRF (Park et al. 2021b)
NDR (Cai et al. (022), and HexPlanc (Cao & Johnson| 2023] Fridovich-Keh et al. (023) - as well as recent ciinespace-partttonni 2023d) (with default training configurations). Please see the Sup. Mat. for further details.
methods Tensor4D (Shao et al. (023)) and NeuS2 (Wang et al.
Insights. We report all quantitative and qualitative results in Tab. and Fig. Results demonstrate that our method consistently improves all baseline methods, achieving new state-of-the-ar results for sparse multi-view reconstruction of dynamic scenes. We further observe that more ResField layers gradually improve results until the point of saturation (i = 1, 2, 3). This experiment con- firms that increasing the modeling capacity more-thane thoe-th level does not cause overfitting. Importantly, the simplest/cheapest baseline method TNeRF implemented with ResFields performs better than every other more expensive baseline method in the original form. We believe that such speedup and lower memory requirements are of great benefit to the research community, as they enable the use of lower-end hardware for sideliiylity reconstructions. Given this observation, we set up a simple camera rig and captured longer and more complex sequences to better understand the limitations.
Lightweight capture from three RGBD views. We capture four se- quences (150 frames) via synchronized Azure Kinects (three for reconstruction and one for validation) and compare TNeRF (w. depth supervision). a baseline with a good balance between computational complexity and accuracy, and its enhancement with ResFields applied to all middle layers. Quantitative evaluation in terms of mean SSIM and
Table 4: Lightweight capture from three RGBD views.
Moon
Bank
Glasses
Hand
Writing
LPIPS. SSIM
LPPSS
SSIM
LMMS
SSIM
LPPPS
SSIM
LPIPS SSIM
TNeRF +ResFiclds
0.234
79.16
0.323
68.85
0,206
80.44
0.239
81.30
0.203 80.00
0.284 70.84
0.164
80.65
0.210 82.09
0.155 86.43
LPIPS (Zhang al. 2018) reported in Tab. demoosttates that ResFields consistently benefits the reconstruction (see visuals in Fig. and the Sup. video). However, we observe that both methods struggle to capture thin and tiny surfaces such the cord of sunglasses.
Table 5: Scene flow.
it's type fud/bord (
4.4 SCENE FLOW
Rel.1. MLP +ResFields
6.88 7.31
offset
3,85 3.85
Scene flow models 3D motion field for every point in space x and time f. We take the same four Deforming Things sequences from
ReLU MLP
T.57 /7.58
16.5 SF(3)
+ResFields
2.64 2.56
RRRL MLP
/.19
Sec 4.2 and learn bi-directional scene flow. We use 80% of tracked mesh vertices to learn the flow and the remaining 20% for evalua- tion. As supervision,
+RessFields
2.18 2.19
RcLU MLP
6.3076.444
use I error between the predicted and the ground truth flow vectors. We consider three motion models that predict 1) offset vectors (Prokudin al. 2023
+ResHfelds
offset
4.43/4.59
ReL. MLP
T.00 /3.T
5.6 (SF(3)
+ResFields R MLP
2.88 / 2.84
Li et al. 2021b), 2)
2.47 /7.48
+ResFieldy
DXTT
1.79 L.79
8

---

Published as a conference paper at ICLR 2024
SE(3) transformation Park et al., 2022aa lat Wang al. 2023a), and 3) coefficients of the DCT bases Wang et al. 2021a Li et al. ).23). For all of them, we consider the same architecture from (Wang 2021 : 8 l-yyyer ReLU-MIL with positional encoding.
ct al.
Insight. Tab. reports the 1 error (scaled by 10²) for the evaluation points. ResFields greatly
benefits learning scene flow across all settings. Moreover, the architecture of 256 neurons is less powerful than its much smaller and faster counterpart (128 neurons) with ResFields (rank of 10) for both forward and backward flow while being around three times faster: 16.5 vs. 5.6 train it/s.
Table 6: ResField modeling.
4.5 ABLATION STUDY
Meon PSNR1 test
ResField modeling (Tab. 6). Residual connections
Siren-512
on the layer weights (W;+ +% (t))t) are more power-
(1, x.) ( Xx b.)
31.89 32.13
+outpui residial weightts (Karms et 20200
ful compared to modeling residuals on the layer out- put that commonly used for conditional generation
d (t. x. ) a. (.W.
b:j m
W
32.84 33.12
+modulated weights Mehua
2011
(Karras et al. (2020), directly modulating layer weights
d (t, x. a. (Wro W.w.o.
waiket b.)
32.65
direct bio 2022 w.to
(W; 3 (.()) (Mehta et al. 2021), or using time- dependent weights (L)
de (1, = W. +b)
35.17
35.95
LoE (Hao al. 2022). Tab. d summarizes the results of these varia- tions on the video approximation task from Sec. 4.1
ResFields
(1 ([ +.O0))x 39.21 39.97
Table 7: Factorization techniques.
Factorization techniques (Tab. 7). We compare our
#params Mean PSNR1 [M]
factorization (Eq. 4 with alternative techniques: no
Factorization
Rank
test
train
factorization (Reiser et al. 2021I. low-rank matrix-
Siren
0.8
31.96
32.29
matrix decomposition (v(t) E RR M
None
236
38.52
48.46
RRxMM.) regressing network parameters (Ha et al. 2017), hierarchical Levels-of-Experts (LoE) (Hao et al. 2022), and the classic CP (Carroll & Chang 1970) and Tucker (1966). CP and Tucker with vary- ing ranks demonstrate good generalization and over- fitting results. No factorization achieves great train- ing PSNR, but its generalization performance is sub- optimal which has been mitigated by the hierarchi- cal formulation of LOE. The proposed factorization achieves the best generalization properties. The re- ported numbers in Tab. 7 are measured on the video approximation task for 30% of unseen pixels. See the Sup. Mat. for additional comparisons.
Resser et 2021
Low
3
3572T
36.53
matrix-matrix:
20
10.0
35.88
37.50
v(C) E M
36.67
39.01
E RRXM
80
37,8
37,69
41.07
Netwer
10.6
38.60
39.56
Ha al
2017
10
0.8
35.04
33.56
20
33.47
Camoll & Chang
1970
40
1.0
33.41
33.75
+ReFFillss
80
1.1
33.72
34.08
10.64.64
1.1
33,96
34.31
40.64,64
1.5
34.67
35.10
Tucker 1966
80,64,64
2.0
35.08
35.59
10,256,256
3.6
36.31 36.90
40,256,256
9.5
38.31 39.31
80,256,256
17.4
39.04 40.39
(2.4.8)
4.5
36.42 37.37
LoB
(8.16.32)
15.5
39.87
42.27
Limitations. Overall ResFields benefits spatiotempo- ral neural fields when the bottleneck lies in the model-
Hao ct al. 0023
(16,32,64)
30.2
40.53
44.15
(32,64, 128)
59.5
40.62
46.35
10
8.7
39.59
40.80
ing capacity rather than in solving unconstrained prob- lems. Specifically, we do not observe an advantage on challenging ill-posed monocular reconstruction (Gao)
Ours
20
40.87
Bq
40
32.3
41.69
43.72
80
63.8
41.51
44.39
cet al. 2022) when the main bottleneck is the lack of constraints rather than the network's 's capacity.
DISCUSSION AND CONCLUSION
We present a novel approach to overcome the limitations of spatiotemporal neural fields in effec- tively modeling long and complex temporal signals. Our key idea is to incorporate temporal residual layers into neural fields, dubbed ResFields. The advantage and utility of the method lie in its ver- satility and straightforward integration into existing works for modeling 2D and 3D temporal fields. ResFields increase the capacity of MLPs without expanding the network architecture in terms of the number of layers and neurons, which allows us to use smaller MLPs without sacrificing the re- construction quality while achieving faster inference and training time with lower GPU memory requirement. We believe that progress towards using lower-cost hardware is the key to democratizing research and making technology more accessible. We hope that study contributes to develop- ment of neural fields and provides valuable insights for modeling signals. This, in turn, can lead to advancements in various domains, including computer graphics, computer vision, and robotics.

---

Published as a conference paper at ICLR 2024
Acknowledgments and Disclosure of Funding. We thank Hongrui Cai and Ruizhi Shao for pro- viding additional details about the baseline methods and Anpei Chen, Shaofei Wang, Songyou Peng, and Theodora Kontogianni for constructive feedback and proofreading the manuscript. This project has been supported by the Innosuisse Flagship project PROFICIENCY No. PFFS-21-19.
Ethics Statement. In our pursuit of advancing signal modeling and representation techniques, our work holds the potential to bring positive advancements to various domains within the entertainment and AI industries, benefiting both research and practical applications. However, crucial to acknowledge the indirect influence of our efforts on the field of deep fakes, our methodology contributes to the enhancement of photorealistic reconstruction from images.
Reproducibility. In our commitment to promoting openness and transparency in research, provide comprehensive resources for replicating our results: 1) Open source code: we will release the source code used in experiments, which can be found in the supplementary material. This code includes detailed documentation and instructions to facilitate the replication of our main results. 2) Pre-trained models: we will release our trained models to improve verifiability. 3) Dataset: our captured dataset will be made publicly available. 4) Supplementary documentation: in addition to the code, we provide supplementary document that offers deeper insight into our experimental setups, training techniques, and other crucial details.
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. Proceedings of the IEEE conference on computer vision and pattern recog- nition workshops, 2017.
the
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Bruale and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang, and Juyong Zhang. Neural surface recon- struction of dynamic scenes with monocular rgb-d camera. In NeurlPS, 2022.
Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes. CVPR, 2023.
J Douglas Carroll and Jih-Jie Chang. Analysis of individual differences in multidimensional scaling via an n-way generalization of "eckart-young" decomposition. Psychometrika, 1970.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awen 3d generative adversarial networks. In CVPR, 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. fields. In ECCV, 2022.
Tensorf: Tensorial radiance
Misha Denil, Babak Shakibi, Laurent Dinh, Mare' Aurelio Ranzato, and Nando De Freitas. Predict- ing parameters in deep learning. NeurIPS, 2013.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.1433 14314, 2023.
Robert A Drebin, Loren Carpenter, and Pat Hanrahan. Volume rendering. ACM Siggraph Computer Graphics, 1988.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. In ICLR, 2020.
Sara Fridovich- Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
10

---

Published as a conference paper at ICLR 2024
Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Dynamic novel- view synthesis: A reality check. In NeurlPS, 2022.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometrie regu- larization for learning shapes. In ICML., 2020.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetoorks. In ICLR, 2017.
Zekun Hao, Arun Mallya, Serge Belongie, and Ming- Yu Liu. Implicit neural representations with levels-of-experts. NeurIPS, 2022.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.
Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. INT J UNCERTAIN FUZZ, 1998.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, and Matthias Nießner. Humanrf: High-fidelity neural radiance fields for humans in motion. ToG, 2023.
Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. arXiv preprint arXiv:1511.0674 2015.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. NeurlPS, 2021.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyz- ing and improving the image quality of stylegan. In CVPR. 2020.
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. In ICLR, 2021.
Taehwan Kim and Tülay Adal. Approximation by fully complex multilayer perceptrons. Neural computation, 2003.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d video synthesis from multi-view video. In CVPR, 2022.
Yang Li, Hikari Takehara, Takafumi Taketomi, Bo Zheng, and Matthias Nießner. 4dcomplete: Non- rigid motion estimation beyond the observable surface. In ICCV, 202
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021b.
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In CVPR, 2023.
David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. In CVPR, 2022.
Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. ECCV, 2022.

---

Published as a conference paper at ICLR 2024
William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics. 1987.
Iyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0333 2016.
Qianli Ma, Jinlong Yang. Siyu Tang, and Michael J. Black. The power of points for modeling humans in clothing. /CCV, 2021.
Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. In ICCV, 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ToG, 2022.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization deep learning. arXiv preprint arXiv: 1412.6614, 2014.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Gold- man, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representa- tion for topologically varying neural radiance fields. ToG, 2021b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NeurlPS, 2019.
Sida Peng, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Representing volumetric videos as dynamic mlp maps. In CVPR, 2023.
Sergey Prokudin, Qianli Ma, Maxime Raafat, Julien Valentin, and Siyu Tang. Dynamic point fields. 2023.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In ICML, 2019.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural
radiance fields with thousands of tiny mlps. In ICCV, 2021.
Frank Rosenblatt. Principles of neurodynamics. perceptrons and the theory of brain mechanisms. Technical report, Comell Aeronautical Lab Inc Buffalo NY, 1961.
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decompasition for high-fidelity dynamic reconstruction and rendering. In CVPR, 2023.
Shayan Shekarforoush, David Lindell, David J Fleet, and Marcus A Brubaker. Residual multiplica- tive filter networks for multiscale reconstruction. NewrIPS, 2022.
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. NewrIPS, 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. NeurIPS, 2020b
12

---

Published as a conference paper at ICLR 2024
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.003887 2015.
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek
Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. of detail: Real-time rendering with implicit 3d shapes. In CVPR, 2021.
Neural geometric level
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representa- tions. In CVPR, 2021.
Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, al. Ad- vances in neural rendering. In Computer Graphics Forum, 2022.
Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 1966.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2105.05990 2021a.
Chaoyang Wang, Lachlan Ewen MacDonald, Laszlo A Jeni, and Simon Lucey. Flow supervision for deformable nerf. In CVPR, 2023a.
Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Morpheus: Neural dynamic 360 surface re- construction from monocular rgb-d video. arXiv preprint arXiv:2312.00778 2023b.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021b.
Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In ICCV, 2023c.
Shaofei Wang, Marko Mihajlovic, Qianli Ma, Andreas Geiger, and Siyu Tang. Metaavatar: Learning animatable clothed human models from few depth images. NeurlPS, 2021c
Yiming Wang, Qin Han, Marc Habermann, Kostas Daniliiiis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In ICCV, 2023d.
Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. Pet-neus: neural surfaces. In CVPR, 2023e.
Positional encoding tri-planes for
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Gruphics Forum. Wiley Online Library, 2022.
Yi Xu, Yao Lu, and Ziyu Wen. Owlii dynamic human mesh sequence dataset. ISO/EC JTC1/SC2//GG11 m41658, 120th MPEG Meeting, 2017.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. NeurIPS, 2021.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Ellen D Zhong, Tristan Bepler, Joseph H Davis, and Bonnie Berger. Reconstructing continuous distributions of 3d protein structure from cryo-em images. arXiv preprint 2019.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
We provide additional implementation details and experiments to complement our study. All the reported runtime in this paper is measured on an NVIDIA RTX 3090 GPU card.
A.1
IMPLEMENTATION DETAILS
Initialization. For experiments that Siren networks (Sitzmann al. 2020b) (sections and
4.2) we follow their proposed initialization scheme. Models used for SDF-based dynamic radiance
hield reconstruction (Sec. 4.3) are initialized following the geometric initialization scheme (Gropp
ct al. 2020). Other neural network weights are initialized following Glorot & Bengio (2010).
Residual weights. Parameters (V,, M.) which model our residual weights are initialized with normal distribution (O, 10-2) to ensure a negligible modification of the initial MLP weights. We observe that larger initial values may negatively affect geometric and Siren initialization. For all experiments in the main paper, we set the number of coefficients Tj to the number of frames unless specified otherwise.
Training details. All models are trained with the Adam optimizer (Kingma & Ba 2015) with default parameters defined by the PyTorch framework (Paszke et al.] 2019). We observe stable convergence with the learning rate of 5 x 10 and gradual cosine annealing (Loshchilov & Hutter 2016) until the minimum learning rate of 5 10-1 for the experiments on dynamic neural radiance fields (Sec. 4.3) For other experiments (sections 4.1 and 4.2), we use the learning rate of 5 x 10-3 and cosine annealing until x 10 All methods are trained respectively for 100, 2 x 10 X 10S, and 6 10a iterations the 2D video approximation task (Sec. 4.1), temporal SDF reconstruction (.ec.4.1)
(Sec. 4.2 and dynamic volumetric reconstruction (Sec. 4.3) on Owlii (Xu et al. 2017) and our captured sequences. An exception in Sec. is with the iet al. 2023) and HexPlane (Cao & Johnson 2023 FFidovichh-Kar et al. (023) which were trained for fewer iterations (2 10") they use shallower MLPs which converge faster.
methods Tenso4D (Shao)
Table A.I: Number of factors.
A.2
2D VIDEO APPROXIMATION TASK
Factors
Mean PSNR
All methods presented in the paper (Sec. 4.1 on the 2D video approximation task are trained for 100k iterations, each iteration containing 200k random samples from the training set.
Siren-S12
100%
39.36
40.13
0556
39.90
40:77
40.69
For the NGP baseline in Sec. 4.1 we follow the default setup and use two-layer fully fused network with ReLU activation func- tions and run grid search of hyperparameters to find the optimal configuration. Specifically, we vary the table size T and the num- ber of levels in Tab. A.3] and found that the best results are achieved with T = 23 and L 8 which is reported in the main paper (Fig. 4). Furthermore, we provide results (Tab. AA55) of a much larger layerlayer Siren network with 1700 neurons per layer to match the number of trainable parameters to ResFields imple- mented with five-layer Siren network with 512 neurons, each containing 8.7M parameters. Expectedly, we observe that both methods achieve similar fitting and generalization performance. However, training such a huge MLP with 1700 neurons becomes impractical, making our approach over six times faster train while requiring over two times less GPU memory.
+ResFields
60%
70%
39.53
30.00
40.49
40.44
30%
39.10
40.04
20%
38.87
39.82
Table A.2: Time interpolation.
Factors
Mesu PSNR
Siren-S12
36
50
21.61
40.90
80 %
22.01
40.82
40.76
+ResField
50%
26.12
40.58
40 %
25.54
40.41
39.91
10:1 %
27.34
39.37
Number of factors (Tab. A.I) We further ablate the impact of the number of factors T; of ResFields. In this experiment, we leave out 10% of randomly sampled pixels for validation and vary the number of factors used for parameterizing the coeffi- cients v E T x R. in particular, we set T; as the percentage of the total number of frames. The results averaged over two videos are reported in Tab, A.I We observe that the best performance is achieved for 95% when there's little overlap between the co-
Table A.3: Layers vs. rank.
ResField Layers
Rank
Aparams [M]
Meon
PSNR1
Ri
test
train
2
1,2.3
13
5
47
38.01
37.53
38.67
58.25
1,2,3
30
10
8.7
39.86
35.75
40.73
2
45
15
12.6
39.35
40
2
60
16.5
39.67
20.88
1.2.3
20
41,20
42.34

---

Published as a conference paper at ICLR 2024
Table A.4: Ablation study of different fractions of unseen pixels on the video approximation task.
ResFields consistently demonstrate good generalization properties regardless of the difficulty level.
Unseon pixels
Meon PSNRT
Cat PSNRT
Bicycles PSNRT
test
train
lest
train
test
train
Siren-512
32.02
32.27
31.21
31.41
32.84
33.13
ResFields
38.58
Siren-1024
10%
36.67
37.36
34.95
35.52
38.38
39.19
ResFields
43.15
44.75
42.49
43.53
43.82
45.98
Siren-512
31.99
32.27
31.18
31.41
32.79
33.13
ResFields
39.74
40.75
38.50
39.15
40.98
42.35
20%
34,90
ResFields
42.95
44,82
42.30
43.53
43.59
46.12
Siren-512
31.97
32.3
31,15
31.42
32,80
33.18
ResFields
39.59
38.39
39.17
42.43
Sirea-1024
30%
36.51
37:42
34.83
35.58
38.19
39.27
RRssFields
42.16
43.60
Sirea-312
31.91
32.29
31.10
31.41
32.71
33.17
ResFields
40%
39.39
40.85
38.28
39.20
40.5
42.50
Siren-1024
36.41
37.50
34.74
35.63
39.38
Restiells
42.33
45.14
41.86
43.67
42.79
46.62
Siren-512
31.85
32.31
31.05
31.42
53.20
ResFields
50%
39.09
40.95
38.08
39.24
40.10
42.66
Siren- 102e
36.26
37.61
34.62
35.71
37.90
39.51
ResFields
Siren-512
31.59
32.40
30.83
31.48
32.35
33.32
ResFields
37.70
41.49
37.14
39.42
38.26
43.55
70%
Siren-1024
35.54
38.04
34.08
36.06
36.99
40,02
ResFields
38.96
46.61
39.00
44.44
38.91
48.78
efficients. In practice, there is a negligible difference compared to using independent coefficients default configuration as it is slightly computationally faster.
(100%) which
Time interpolation (Tab. A.2). One downside of using per-frame independent coefficients that it does not support time interpolation. We conduct experiment to evaluate the interpolation along the time axis. Here we randomly sample 10% of frames and leave them out for validation. As expected, the lower number of factors T; leads greater overlap among the frames, consequently leading to better interpolation properties results, while gradually decreasing the training PSNR.
Layers vs. rank (Tab. A.3). Another natural question to consider is whether is more beneficial to have more ResField layers or single ResFied layer with a higher rank while maintaining the constant number of trainable parameters. We conduct this experiment on the video approximation task and compare methods with an equal number of parameters. We conclude that multiple ResField layers provide greater modeling capacity.
Ablation for fewer training samples (Tab. A.4). To complete our study and better understand the implicit bias of our method, we further benchmark Siren and ResFields with varying difficulty levels, ranging from 10-70% of unseen pixels. We observe that ResFields consistently demonstrate good generalization across all the levels of difficulty, well above the baseline.
A.3 TEMPORAL SIGNED DISTANCE FUNCTIONS (SDF)
The architecture of the Siren MLP used for this experiment is identical to the used for the video approximation task. All methods are trained for 200k iterations, each batch containing 200k samples uniformly sampled across time. For each frame we follow the sampling strategy for static SDFs (uüleer et al. 2022 and sample 50% of points on the mesh 37.5% normally distributed around the surface [00 2) and remaining 12.5% are randomly sampled space.
We provide the full breakdown of per-sequence results in Tab, T.b.
A.4 TEMPORAL NEURAL RADIANCE FIELDS (NERF)
All methods on the Owlii dataset are trained for 400k iterations, except HexPlanes (Cao & Johnson 2023 Fridovich-Kei Keil et al. 2023) which converges faster (400k iterations) due to using shallower MLPs, and Tensor4D and NeuS2 for which we follow the default training scheme as the methods require particular training strategy. The main baselines (TNeRF, DyNeRF, DNeRF, Nerfies, Hy-
2

---

Published as a conference paper at ICLR 2024
Table A.5: Extended ablation study the video approximation task. Siren+ResFields with 512 neurons and Siren with 1700 neurons have an equal number of parameters, however, optimizing huge MLP with 1700 neurons comes with great computational cost. Our method achieves six times faster training while requiring over two times less GPU memory.
Resourves
Mean
Cat Video
Bikes Video
[itA]]
GPU [G]
fiparams [M]
test PSNR
train PSNR
lest PSNR
train PSNR
test PSNR1
train PSNR
Siren-512
+R *ResFields
R=10
9.78
6.5
8.7
39.21
39.97
37.96
38.44
40.46
41.50
Siren-1 1700
1.42
15
8.7
39.15
40.20
37.26
38.14
41.04
42.25
1=7
153
1.3
5.7
32.08
35,06
30.89
33.42
33.28
36.70
NGP Tw20
31,34
35.15
33.88
38.13
L=9
158
1.1
9.9
32.41
37,80
31,06
36.21
33.75
39.39
L=6
126
166
5.1
32.02
34.48
30.51
32.20
33.52
36.76
NGP
Ta8
146
1.3
13.5
33,66
39.33
31.97
37.29
35.34
41.37
L=9
157
1.2
17.7
33.53
31.89
35.17
43.34
1=6
101
2.1
32.01
34.47
30.51
32.20
33.51
36.73
T=22
L=7
116
1.7
13.5
33.39
37.82
31 31.82
35.39
34.96
40.25
L-9
156
1.2
30.3
33.73
43.83
32.17
41.69
35.29
45.96
1=6
75
2.7
5.1
32,02
34.48
30.52
32.20
33.52
36.76
T-23
1-7
99
2.1
17.4
33,83
39.51
32.14
36.44
35.52
42.57
L-8
131
1.6
34.2
34.52
43.85
32,91
40.85
36.13
46.85
35,28
L=6
51
3.8
32.01
34,46
30.51
32.20
33.51
36.72
NGP4
1=7
77
2.7
17.4
33,84
39.51
32.14
36.45
35.54
42.58
L-8
130
1.6
51.0
34.27
44.68
32.72
41.71
35.81
47.66
L=9
145
1.2
84.5
34.02
49.51
33.08
46.58
34.95
52.43
Table A.6: Temporal signed distance function. Siren implemented with our Residual Field layers consistently improves the reconstruction quality. Moreover Siren with 128 neurons and ResFields (rank 40) performs better compared to the much bigger vanilla Siren with 256 neurons. Hence lead- ing to over two times faster inference and convergence time while maintaining lower GPU memory requirements: {[mss] denotes the average inference time for one million query points. Note that using higher ranks almost does not affect the overall time complexity as the main bottleneck is the total
number of queries and neurons.
Resourees
Mean
Bear
Tiger
Vampire CDJ
Vanguard CDJ
ReSynth CD}
GPUL {[ms]]
CD}
NDJ
CDJ
NDJ
CDJ
NDJ
NDJ
NDJ
NDJ
2.4G
15.063
27.23
4.519
4,934
63.686
42.953
15.960
+ TesFiildsTranneeeee
9.477
T8.337
6883
3,365
4,422
3,63
T2.T05
39.457
[1.200
30.414
12,504
13.606
ResFields (rank=20)
ResFields (rank=10)
2.50
20.25
8.427
8.785
16.608
15.483
6.659
6.671
3.301
3.350
4.325
4.351
3.328
3.435
8.708
9.545
32.220
29.661
11.140
10.465
29.961
27.541
11,980
12.216
14.075
13.584
+ ResFields (rank=40)
8.188
14.195
3.201 6.579
4.278
3.148
8.563
29.064
9,729
23.564
11.640
11.999
Siren-256
3.6G
16.373
3.115
3,055
36,666
12.004
ResFields (5mnk:55)
7.901
13,000
6.430
3.009
4.177
2,854
8.576
28. 846
8,993
20.168
11.331
10.124
* ResFields (rank=10)
7.714
12,242
6.408
3.001
4.161
2.814
8.136
27.404
8.757
18.950
11.110
9.040
ResFields (rank=20)
3.8G
48.19
11.840
6.396
4,141
2.753
8.076
27.013
18.056
11,050
8.385
ResFields (rank=40)
7.675
11.674
6.381
3.070
4.137
2.723
8.243
27.296
8,525
17.526
11.087
7.755

---

Published as a conference paper at ICLR 2024
PerNeRF. and NDR) are implemented with the MLP architecture from VoISDF Yariv al. 2021
however, we reduced the original MLP size from 256 to 128 neurons as it is impractical to train such large MLPs more expensive multi-view temporal sequences. The SDF MLP is followed by a two-layer color MIP that takes the output feature of the SDF network. Different from the origi- nal color MLP that is conditioned on the viewing ray direction, we do not pass this information to the network as it is impossible to capture any view-dependent appearance for this extremely sparse setup, We observe that training without the viewing direction stabilizes training of the baselines, especially those that rely on a deformation network. We follow the original formulation of the flow MLPs used in DNeRF, Nerfies, HyperNeRE, and NDR.
Note that the original formulation of the HexPlane method (Cao & Johnson 2023 Fridovich-Keil et al., (023) is not suited for reconstruction from sparse views mainly due to the lack of spherical initialization. Hence we adopt the geometric initialization developed for Triplanes proposed in PET-NeuS (Wang ct al. 2023e). Unlike the other MLP-based methods, the SDF MLP head of HexPlane has four linear layers with ReLU activation functions. Furthermore, to reduce the grid-like artifacts caused by space-time time discretization,
tune the resolution of planes (128 grid locations per dimension) and employ the addi- tional total variation loss akin (Cao & Johnson 2023). We observe that using a higher resolution for six planes impairs the reconstruction quality.
To make the comparison fair among the baselines, for rendering we employ non-biased uniform sampling along the ray and sample 1100 rays during the training. On each ray, we sample 256 points where the starting and exiting points of the ray are calculated by ray-box intersection. The box for each se- quence is estimated from the ground truth scans with a small padding of 5%. All the methods are supervised by minimizing the loss term in Eq. 5 where we set X and A2 to 0.1.
In practice, the SDF- density formulation performs better under sparse setup due to well- rell-behaved surfaces. However, for completeness, we repeat this experiment with the original NeRF formulation (Tab. [A.]). The results demon- strate that all of the baselines consistently benefit from ResFields, making Ner- Rees+ResFields the overall best perffrrnnn method in terms of geometry and HyperNeRF++ResFields in terms of appearance.
Figure A.1: Base MLP weights.
Interpreting residual weights. To better understand the internal workings of ResField MLPs, extract a mesh from the base MIP weights without the residual parameters Wx(t) - see Fig. A. for results on the basketball sequence. The extracted mesh demonstrates that the base MLP successfully discovers a pattern that is shared among all frames.
A.5 PRACTICAL RECONSTRUCTION FROM A LIGHTWEIGHT CAPTURE SYSTEM
We follow the VolSDF architecture from the experiment on the Owlii dataset and train all methods for 600k iterations (2200 rays per batch) since the sequences are longer and images are of higher resolution x640
As the scenes are forward-facine and not constrained from the opposite side, we employ the depth (Cai et al.) 2022) and the sparseness loss Lsparse (Long et al. 2022). In sum, the final loss term is:
Leoter + Cappth + X2 + > >
(A.1)
Figure A.2: Data capture. where we set '1, A2. and Ag to 0.1 and activate Laparse after 70k iterations.
Data capture. Inspired by the lightweight practical capture of surgeries]] we create an identical camera rig with four cameras (see A.2), three for training and one for evaluation. Additionally, we crop captured images by projecting an approximated 31) bounding box around the dynamic region as the main focus of our approach is reconstructing dynamic sequences.
OR-X Setup
4

---

Published as a conference paper at ICLR 2024
Table A.7: Temporal radiance field reconstruction on the Owlii dataset (Xu et al. 2017) with the NeRF parametrization. Previous state-of ofthe-oft methods consistently benefit from our residual field layers without the computational overhead; results with the VoISDF parametrization are provided
in Tab. 3 Colors denote the overall 1st,
2nd and 3rd best-performing model; denotes which
layers are substituted with ResFeilds layers (Ra 10).
Basketball
Model
Dancer
Exercise
CDJ SSIM? PSNKT
CDL
SSIM PSNKT
CDL
SSIMY
PSNKT
CD)
SSIM?
PSNK?
CDJ
SSIM
PSNK"
TNcRF et 2022
61.6
93.90
26.04
70.9
94.06
25.88
52.9
93.18
27.06
65.4
93.49
25.22
57.1
94.88
25.99
ResFields D
53.6
94.54
26.72
53.8
95.08
27.15
53.4
93.55
27.32
53.3
94.40
26.22
54.0
95.14
26.20
ResFields
46.1
27.00
44.6
95.23
27.20
49.6
93.92
27.66
26.80
42.3
95.28
26.32
Restields (1-
1.....7 7)
47.6
95.03
27.16
49.6
95.42
27.51
46.3
94,29
27.96
43.5
94.96
26.74
50.9
95.43
26.44
DyNeRF al
2022
60.8
93.68
25.78
60.9
94.03
25.89
56.2
92.60
26
70,1
93.46
25.26
56.1
94.63
25.79
ResFictas 11=
52.6 94.25
26.42
55.9
94.59
26.70
55.5
93.23
27.08
50.5
94,22
26.13
48.4
94.95
25.75
ResFields (i
1,2,
51.8 94.42
26.53
51.3
94.94
26.74
55.3
93.31 27.07
48.8
94.40
26.33
51.6
95.06
25.99
ResFields
48.9 94.60
26.72
51.5
95.04
26.95
55.7
93.63
27.40
45,7 94.50
26.23
42.8
95.22
26.28
DNGRF
138.3 92.54
24.40
128.8
92.90
24:34
92.1
91.19
25.04
191.5
92.35
23.50
[40.7
93.76
24.72
ResFiclds (i=
56.5 94.36
26.47
48.0
95.05
27.03
63.8
92.89
26.35
61.9 94.29
26.09
52.2
95.19 26.41
ResFields (1
49.6 94.59
26.68
49.3
95.23
27.14
64.0
93.07
26.58
42.2
94.87
26.75
42.8
95.18
26.22
ResFields (i
52.7 94.81
26.88
47.5
95.32
26.90
61.6
93.44
26.98
49.0
94.99
26.87
95.47 52.5
26.77
Netties Purk
2021a
135.0
25:21
26.20
24.73
94.23
25.26
Resd 11
52.0
94.75
26.99
48.0
95.16
27.14
51.7
93.96
27.79
55.3 94.59
26.36
53.1
95.29
26.66
ResFields (1
1,
45.3
94.80
26.88
41.7
95.18
26.70
50.4
93,99
27.82
42.7
94.72
26.49
46.4
95.31
26.52
ResFields
42.2
94.90
26.73
51.8
95.31
26.71
23.6
93.84
26.97
43.6 95.06
26.76
49.8
95.41
26.49
HyperNoRF Purk al 20216
63.5
94.67
26.51
59.9
94.64
26.01
57.8
94.25
27.55
69.7
94.44
25.75
66.7
95.35
26.74
ResFiclds
46.9
41.0
27.14
50.4
27.70
45.6 94.73
ResFFellds (i
2,2,3)
47.1 94.85
26.99
40.7
95.40
27.41
46.5
93.90
27.71
49.2
94.89
26.75
$2.0
95.20
26.07
ResFields (1
7.....7)
48.0 995.07
27:27
50.7
95.46
27.50
49.5
94.14
27.90
95.22
27.03
26.66
NDR at.
66.2 94.50
26.48
64.1
94,84
26.44
55.8
27.40
78,1
93.93
25.34
06.8
95.17
26.53
ResFields 1s
49.5
94.71
26.89
50.4
95.02
26.90
51.8
93.85
27.64
47.6
94.67
26.48
48.3
95.32
26.53
ResFields 2, 30
27.13
46.2
27.36
51.0
27.62
94,91
26.85
95.42
26.69
ResFields (1 ......7) .7
49.8
94.97
27.08
44.2
95.31
27.14
50.7
94.20
27.82
49.3
95.03
26.87
55.2
95.37
26.52
Table A.8: Capacity of the low-rank representation. We measure the capacity of the low-rank videos (250 frames) with varying levels of difficulty: /) a video composed of randomly selected images (250 segments), 2) video consisting of 6 coherent segments, and 3) full video depicting one sequence. We observe that increasing the number of in- dependent segments has a a significant effect on the model performance due to the lack of information that is shared across the entire signal and could be encoded in the base matrix weights. In spite of this, our method successfully improves quality over the vanilla Siren even in the most challenging case (29.18 vs. 19.4 PSNR for rank 40).
parametrization on 351x510-resoluth
Random Video
Bikes Video
Cat Viden
Meam
(250 segments) train PSNR
(6 segments)
(1 aegment) train PSNR
Rank
PSNR
Irain PSNR
PSNR
PSNR
train PSNR
PSNR
Siren
29.86
30.21
19.40
19.65
33.68
34.05
36.49
36.94
$2.99
$3.52
23.54
23.18
36.85
$7.40
38.77
30.58
-ResFields
34.18
34.89
24.26
24.89
38.26
38.98
40.02
40.80
35.83
36.82
25.51
26.47
39.95
40.94
42.02
43.06
37.31
38.73
26.84
28.39
42.82
43.62
10
37.73
39.33
27.26
29.09
41.86
43.34
44.07
45.57
38.74
41.08
28.39
42.82
45.01
40
39.43
42.54
29.18
33.64
43.32
45.85
45.78
48.12
A.6 ADDITIONAL INSIGHTS
Capacity of the low-rank representation. Learning signals with many independent parts imposes additional challenges for both ResFields and coordinate MLPs in general. The coordinate MLP weights possess the ability to compress spatio-temporal signal into a compact representation, mainly because the source signal has sigmificant amount of repetitive information which is ef- fectively represented by the optimized network weights. The shared weights of the ResFields have the same property, as they tend to store information that is common across the entire signal, whereas the low-rank weights accommodate for topological and dynamic changes for effective learning.
We conduct the following experiment to analyze the learning capacity of our low-rank weights in the case of dynamic scenes with varying complexity. We create three videos with different levels of difficulty: 1) A corner case when every frame depicts novel scene (a video composed from the first
5

---

Results are reported in Tab. A.8 We observe that increasing the number of independent video segments indeed has a significant effect on the model performance. This can be attributed to the lack of information that can be shared across the entire signal and efficiently encoded in the base matrix weights. However, despite this, our method successfully improves quality over the vanilla Siren, even in the most challenging case of random video (19.4 vs 29.18 PSNR for our model with rank 40). This improvement can he attributed to (a) the increased capacity of the pipeline thanks to residual modeling and (b) the ability of the neural network to discover small patterns that could be effectively compressed into shared weights even in the extreme case of a video with completely random frames. The latter behavior is especially apparent in the simplest case when ResFields uses only rank (19.4 vs 23.34 PSNR for the random video sequence). Please also note that increasing demonstrate in Published as a conference paper at ICLR 2024
250 images from the DIV2K dataset Agustsson & Timofte (2017)) 2) the bikes video containing segments, and 3) the cat video contaiming only segment. All of these three videos are trimmed and cropped to the same length (250 frames) and resolution (351 x 510). Then, perform the 2D video approximation (analogous to Sec. 4.1) by learning the mapping from a space-time coordinate to RGB color (10% of pixels not seen during the training and are left for evaluation).
the capacity in case does not lead significant computational overhead, the main submission and further emphasize in the next section.
Modeling long sequences. ResFields approach faces limitations for long and evolving signals, as the shared weight matrix runs out of capacity. Here, a straightforward strategy to deal with longer sequences utilized in the literature (Li al.. 2022) is to split the sequence into independent chunks and train separate neural fields. We investigate different strategies for addressing the sequences of arbitrary length.
Table A.9: Modeling long sequences.
As our test bed, we consider the 2D video approximation task (Sec. 4.1p with a longer 12x228-resolution video (1024 frames). Here, we re-pupoose the video captured by one of our Kinects for the dynamic NeRF reconstruction task. We divide the video sequence into several chunks of varying sizes: 512, 256, 128, and 64 frames (2, 4, 8, and 16 sub-sequences respectively). We consider maintaining for each chunk a set of shared W; and residual weights (t(t) well both together (W, (t))). For exam- ple, having 4 chunks and both shared and delta weights updated would mean having 4 separate tensors and 4 sep- arate factorized delta weights. The results are summa- rized in the Tab. A.9
#params
Chunking
PSNR1
[
#chunks
test
train
Siren
0.2
28.18
28.21
2.5
34.38
34.78
2.6
shared
35.65
35.87
4.2
2
residual
35.96
36.22
4.6
both
37.01
37.31
AResFields
3.0
stared
37.06
37.32
residual
37.30
8.9
both
38.89
39,31
3.8
shared
38.58
38.90
16.0
8
residual
38,36
38.84
17.5
both
40.49
41.08
shored
39.35
39.56
31.7
16
residual
39.47
40.06
41.32
We observe that as we increase the number of sub-sequences, the results gradually improve for all three strategies, with the best overall quality achieved by independently updating both weights. However, this strategy naturally requires the largest amount of parameters. We noted that the strat- egy of chunking only shared weights is the most parameter-efficient for processing coherent long sequences. Specifically, maintaining 16 shared weights for the whole sequence achieves better qual- ity compared to maintaining sets of all considered network parameters (39.55 vs. 38.89 test PSNR), while using fewer parameters (5.3M vs. 8.9M).


Reviews:
[{"review_id": "8wKMnSQlhR", "reviewer": "Reviewer_4JwP", "summary": "This paper addresses the challenge of modeling non-static scenes with NeRF, where introducing a temporal dimension 't' significantly increases the required model capacity. A method for temporal information modeling is proposed, which by employing a low-rank representation, controls the amount of parameters to be learned. This approach enables temporal modeling without affecting the network size. Alongside, this paper innovates by integrating temporal residual layers in neural fields, dubbed ResFields, showcasing an effective way to represent complex temporal signals without increasing the size of Multi-Layer Perceptrons (MLPs), thus offering a promising solution to the capacity bottleneck for modeling and reconstructing spatiotemporal signals.", "questions": "1. The symbols in Equation 5 are not explained.\n2. Why are the results of NGP plus ResFields not shown in Table 1?\n3. Can ResFields, like LoRA[5], provide further dynamic modeling capabilities to a pre-trained NeRF?\n\n[1]Wang C, MacDonald L E, Jeni L A, et al. Flow supervision for Deformable NeRF[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 21128-21137.\n[2]Li Z, Wang Q, Cole F, et al. Dynibar: Neural dynamic image-based rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 4273-4284.\n[3] Wang Y, Han Q, Habermann M, et al. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 3295-3306.\n[4] Wang Q, Chang Y Y, Cai R, et al. Tracking Everything Everywhere All at Once[J]. arXiv preprint arXiv:2306.05422, 2023.\n[5]Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models[J]. arXiv preprint arXiv:2106.09685, 2021.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Strengths:\nThe method proposed in this paper is simple and effective, with clear and easy-to-understand writing, and ample experiments.\nNovelty:\nThis paper introduces a lightweight, plug-and-play module to enhance NeRF's  capability for dynamic objects.\nGeneralization:\nThis paper applies ResFields to various base models and downstream tasks, achieving a certain performance improvement across the board.", "weaknesses": "The main issue is that this paper lacks references to and comparisons with closely related methods, such as [1][2][3][4]. These works all address the problem of modeling non-static scenes, which is closely related to the theme of this paper. These methods primarily model a normalized space, then model the relationship between the 3D expression at each moment 't' and the normalized space. The absence of this comparison leads to (1) unclear performance advantages, and (2) uncertainty about whether ResFields could also be applied to these methods to further enhance temporal modeling capability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wr2L3nVBMZ", "reviewer": "Reviewer_ZKNQ", "summary": "This paper propose a way to extend MLP-based neural fields to have the capability to model time-dependent signals. The key ideas is to modulate the residual of the MLP weights by a time-dependent matrices decomposed into a vector matrix. This decomposition share a right basis, which allows reusing structures between time steps, providing regularity and potentially helping with generality. The paper shows good quantitative results that support the claim. I believe this idea is simple and effective, and it’s can be a good contribution for the community.", "questions": "See weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is very simple. The simplicity of such idea allows it to be combined with different design of neural fields as long as the main training parameters are parameterized by matrices.\n- The quantitative results shows that it’s also effective to certain degree. The ablation studies also seems to support the effectiveness of techniques.", "weaknesses": "- (minor) The paper provides little intuition of why this particular factorization is chosen rather than some alternative ways to factorize these matrices. For example, can we make v(t) to be N_ixR_i and M_i to be R_ixM_i? Is there any intuitive reason why this is not a good choice? Maybe this is addressed in the ablation study in Section 4.4, but providing some more intuition is good.\n- (minor) While the idea seems to support any matrix-like weight parameterization, the specific factorization might provide different interpretation depending on the way the network use the weight matrix. Maybe the specific design choice is limited to MLP architecture.\n- Concern about long sequence. The sequence weight at time t is modeled as W_o + \\Delta W_t. When t is very far from the original frame, then the weights \\Delta W_t can have significant weights, so it’s not clear to me whether sharing the same matrix M across all time steps is a good idea. Maybe the author can consider periodically updating the weight W_o once a while?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HSseppnVFW", "reviewer": "Reviewer_KYty", "summary": "The paper proposes a method for improving the performance of coordinate-based network representation of various temporally varying signals. The key contribution of the method is that rather than having the network learn to represent temporal variation in a completely unstructured way, the temporal variation is instead modeled by a low-rank weight matrix which is added to a base weight matrix that is shared across all time instances. This exploits the temporal consistency in the underlying signals being represented, and results in a more parameter efficient and high-quality representation of the underlying signal. This is benchmarked across various tasks in which coordinate-based networks are used, such as video overfitting, dynamic SDF fitting, and dynamic neural radiance field fitting, where ResFields (the contribution) demonstrate consistent increased performance.", "questions": "I do not have any additional questions on the manuscript. Overall, as described in the strengths section, I see the paper as a good paper: the method is described well, simple, and makes intuitive sense, and then it is evaluated well across a number of tasks and ablated to show that the improvement comes from the proposed contribution. For this reason I am positive on the paper and leaning towards acceptance. I see some minor weaknesses in the potential capacity of the model, and especially in the potential computational overhead of the method in training speed. Addressing these weaknesses would incline me to raise my scores for the paper.\n\n**Update after author response**\n\nThank you for the detailed response. It has addressed my questions. I have updated my score in accordance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In my opinion the strengths of the paper are as follows:\n1. The paper is described exceptionally clearly, and the method makes intuitive sense on why it would improve performance for representation on temporal signals. The relative simplicity of the method and lack of reliance on a number of complex hyperparameters makes it more likely that future methods will be influenced and use the contributions.\n2. The evaluations are thorough, across three different temporal signal representation tasks, all demonstrating clear improvements from the ResField contribution. In each of these domains, a number of baseline methods are all compared against. I found that the comparisons on dynamic NeRF especially are thorough, using many different state-of-the-art methods and all showing that ResFields leads to increased performance.\n3. The method is ablated well, and various ablations show the contribution of the method is actual source of the improvements.", "weaknesses": "In my opinion, the main weaknesses of the paper are:\n1. To me, it is unclear how much capacity the low-rank representation has to model various dynamic components. While this may not be a problem in something like dynamic NeRF or dynamic SDF where the signal remains relatively constant over time, for overfitting a 2D video, where there may be cuts or completely different scenes, I would be interested to see how the method performs. In this case, it seems like the low-rank weight matrix approximation would be required to model all of the signal, as there is little consistency shared in the underlying signal over time and thus less information to store in the shared weights. Could this potentially be a limitation of the method?\n2. One additional minor weakness is that the method seems like it may add significant amount of computational overhead. For every iteration, the weight matrix of the underlying representation needs to be changed. Does this result in significant slowdown in optimization? It would be an insightful comparison to include this in the baseline comparisons, especially in the dynamic NeRF scenario where speed of fitting is very important. If this adds a significant amount of computational overhead, it is possible that the amount of gains is not worth it, and could be achieved by simply training a standard representation for longer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)