ReMatching Dynamic Reconstruction Flow
Authors: Sara Oblak
Despoina Paschalidou
Sanja Fidler
Matan Atzmon
Venue: ICLR 2025 Poster (2025)
Keywords: Dynamic Reconstruction
Flow Modeling
Gaussian Splatting
Novel view synthesis
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We introduce the ReMatching framework—a novel method for designing and incorporating deformation priors into dynamic reconstruction models, ensuring fidelity to input data while adhering to the specified priors.

Abstract: Reconstructing a dynamic scene from image inputs is a fundamental computer
vision task with many downstream applications. Despite recent advancements, existing approaches still struggle to achieve high-quality reconstructions from unseen viewpoints and timestamps. This work introduces the ReMatching framework, designed to improve reconstruction quality by incorporating deformation priors into dynamic reconstruction models. Our approach advocates for velocity-field based priors, for which we suggest a matching procedure that can seamlessly supplement existing dynamic reconstruction pipelines. The framework is highly adaptable and can be applied to various dynamic representations. Moreover, it supports integrating multiple types of model priors and enables combining simpler ones to create more complex classes. Our evaluations on popular benchmarks involving both synthetic and real-world dynamic scenes demonstrate that augmenting current state-of-the-art methods with our approach leads to a clear improvement in reconstruction accuracy.

Published as a conference paper at ICLR 2025
REMATCHING DYNAMIC RECONSTRUCTION FLOW
Sara Oblak' Despoina Paschalidou Sanja Fidler Matan Atzmon"
NVIDIA University of Toronto Vector Institute
(soblak, beaschalidouy sfidler, mat mann mma.eeeiia.mm.n envidia.com
ABSTRACT
Reconstructing dynamic scene from image inputs is fundamental computer vision task with many downstream applications. Despite recent advancements, ex- isting approaches still struggle achieve high-quality reconstructions from unseen viewpoints and timestamps. This work introduces the ReMatching framework, designed to improve reconstruction quality by incorporating deformation priors into dynamic reconstruction models, Our approach advocates for velocity-field-bar based priors, for which we suggest matching procedure that can seamlessly supplement existing dynamic reconstruction pipelines. The framework is highly adaptable and can be applied to various dynamic representations. Moreover, it supports integrating multiple types of model priors and enables combining simpler ones to create more complex classes. Our evaluations on popular benchmarks involving both synthetic and real- world dynamic scenes demonstrate that augmenting cur- rent state-of-the-ar art methods with our approach leads to a clear improvement reconstruction accuracy.
INTRODUCTION
This work addresses the challenging task of novel-view dynamic reconstruction. That is, given set of images of dynamic scene evolving over time, the task objective is to render images from any novel view intermediate point in time. Despite significant progress in recent years (Lombardi et al., 2022]] Fridovich-Keil et al. 2023 Yunus et al. 2024). effectively learning representations of dynamic scenes still remains an open challenge. The main hurdle arises from the typically sparse nature of multi-view inputs, both temporally and spatially. To address this, prior knowledge is often incorporated either from physical prior such as rigidity (Sorkine & Alexa (007)). or data-driven priors derived from large foundation models (Ling et al.] 2024 Wang et al. 2024). However, existing approaches struggle with an inherent trade-off: while priors improve generalization, they often compromise reconstruction fidelity by failing to match exactly the given input. In turn, designing a method that effectively integrates priors without sacrificing high-fidelity reconstructions remains unresolved.
To address this issue, we introduce the ReMatching framework, novel approach for designing and integrating deformation priors into dynamic reconstruction models. The ReMatching framework has three core features: i) optimization objective that aligns reconstruction solutions with prior regularization as closely as possible, without sacrificing fidelity; ii) ensured applicability to various model functions, including time-dependent rendered pixels particles representing scene geometry; and, iii) provide a flexible design of deformation prior classes, allowing more complex classes to be built from simpler ones.
To support the usage of rich deformation prior classes, we advocate for priors expressed through velocity fields. A velocity field a mathematical object that describes the instantaneous change in time the deformation induces. As such, a velocity field can potentially provide a simpler characterization of the underlying flow deformation. For example, the complex class of volume- preeeee-reeriinggg flow deformations is characterized by the condition of being generated by divergence- free velocity fields (Fisenberger et al. 2019). However, representing a deformation through its generating velocity field typically necessitates numerical simulation for integration, procedure that can be computationally expensive and cone-consuming. Nevertheless, recent progress in flow-based generative models (Ben) Hamu et al. 2022 Lipman et al. 2022 Albergo al. 2023) supports simulation- free flow training, inspiring this work to explore simulation free training for flow-based dynamic reconstruction models. Therefore, our framework is specifically designed to integrate with dynamic reconstruction models that represent dynamic scenes directly through time-dependent reconstruction functions (Pumarola
et al. 2021 Yang et al. 2023).

---

Published as a conference paper at ICLR 2025
Exploiting the simplicity offered by sewecity-field-baee deformation prior classes, we observe that the projection of a time-dependent reconstruction function onto a velocity-fiele prior class can be framed as a flow- fow-matching problem. The opportunity access the projected flow is reminiscent of the Alternating Projections Method (APM) (Deutsch) 1992). a greedy algorithm guaranteed in finding the closest points between two sets. Therefore, we suggest optimization objective aimed at re-projecting back onto the set of reconstruction flows. This corresponds to a flow-matching loss that we term the ReMatthing loss. Our hypothesis is that by mimicking the APM, this optimization would converge to solutions that not only meet the reconstruction objective, but also reach the closest possible alignment to the required prior class. By doing so, we achieve the desired goal of improving generalization to unseen timestamps without compromising solutions' fidelity levels.
We instantiate our framework with a dynamic model based on the popular Gaussian Splats (eerbl et al. 2023) rendering model. We explore several constructions for deformation prior classes including piece-wise rigid and preserring deformations. Additionally, we demonstrate our framework's usability for two different types of time-dependent functions: rendered image pixels color, and particle positions representing scene geometry. Lastly, we evaluate our framework on standard dynamic reconstruction benchmarks, involving both synthetic and real-world scenes, and showcase clear improvement in reconstruction quality.
Our contributions. In summary, the main contributions of this paper are:
We propose the ReMatching framework, which controls the optimization of dynamic recon- struction models converge to solutions that closely align with predefined prior class of deformations. In turn, the framework balances achieving high-fidelity reconstructions with leveraging the benefits of adhering to prior assumptions.
2. The framework unifies various types of model functions, including geometry representations and image rendering, under single cohesive approach, ensuring broad applicability and making future advancements within it relevant to a wide range models.
3. The framework allows for the combination of multiple prior classes, enabling users to design and adapt the method for their specific reconstruction settings.
RELATED WORK
Flow-based 3D dynamics. There is an extensive body of works utilizing flow-based deformations for 3D related problems. For shape interpolation, (Eisenberger et al. 2019) considers volume- preserving flows. For dynamic geometry reconstruction, (Niemeyer et al. 2019) suggests learning
neural parametrizations of velocity fields. This representation further improved by augmenting with a canonicalized object space parameterization (Rempe et al. 2020 Ren et al. 2021) or by simultaneously optimizing for 3D reconstruction and motion flow estimation (Vu al. | 2022).
Similarly to (Niemeyer et al. 2019), (Du et al.) 202T) suggests flow-based representation of dynamic rendering model based on a neural radiance field (Mildenhall al. (2020). More recently, (Chu et 2022, Yu et al. 2023) explores combining time-aware neural radiance field with velocity field for modelling fluid dynamics. In contrast to our framework they focus exclusively on recovering the deformation of specific fluids i.e. smoke and not on reconstructing generic non-rigid objects.
Dynamic novel-view rendering models. Neural Radiance Fields (NeRF) (Mildenhall et al. 2020
a popular image rendering model combining an implicit neural network with volumetric rendering.
Several follow-up works (Pumarola et al. 2021 Park et al. 2021a Tretschk et al. (02T) explore using NeRF for non-rigid reconstruction. by optimizing for dependent deformations. More recently. several works (Fridovich- Keil et al. 2023; Cao & Johnson 2023 Wu et al., 2023 [ong et al.) 2023
Guo et al. 2023) try to address the training and inference inefficiencies of continuous volumetric representations by incorporating planes and grids into spatio-temporal NeRF. An alternative to
NeRF, suggesting an explicit scene representation, is the Gaussian Splatting (Kerbl et al. 2023
rendering model. Several works incorporate dynamics with Gaussian Splatting. (Yang et al. 2023
introduce a time-conditioned local deformation network. Similarly, (Wu et al. 2023) also relies on canonical representation of scene but further improves efficiency by considering a deformation model based on on k- k-planes (Fridovich-Kei) et al. 2023). Lastly, (Lu et al. 2024) propose the integration of global deformation model.
2

---

Published as a conference paper at ICLR 2025
3 METHOD
Given a collection of images, F; {4]) captured at T time steps, from M 1 viewing directions, we seek to develop an image-based model for novel-view synthesis that can effectively render new images from unseen viewpoints in any direction d E 8 and any time t E [[,, tr.. Since we aim to support several time-dependent elements in a dynamic reconstruction model, we employ a general notation for a dynamic image model. That is,
t V (1(1))) -
(I)
with V representing the evaluation at time of all of the model components. Each element function db: R. V. where V is a vector space, can specify any time-dependent quantity specified by the model. V denotes different vector space depending on what và models. For instance, if sb models time-dependent image pixels RGB color, V CR {SIS: : Rd - I& V exists and continuous) with 2. Whereas, if & models the time-dependent position of particles representing the underlying scene geometry, V = R with d = 3. Lastly, in what follows, we interchangeably switch between the notations (1) and 1/+-
The common scheme to learning V, involves supervising the model's image predictions at the given timestamps to reconstruct the input images F.. The specific details of the time-dependent reconstruction model V and the reconstruction loss are deferred to Section [3] We begin first by introducing our proposed framework for incorporating priors through velocity fields.
3.1 VELOCITY FIELDS
We consider a velocity field to be a time-dependent function of the form:
v : Rdd x R2 Rd
where usually d = 3 or d = 2. A velocity field defines a time time-dependent deformation in space Φe RR Rd. also known as a flow, via an Ordinary Differential Equation (ODE):
0
())) (()( Ot d0(x)) = x.
(3)
Flow-based deformations are an ubiquitous modeling tool (Rezende & Mohamed 2015 Chen et al. 2018) that has been extensively used in various dynamic reconstruction tasks (Niemeyer et al. 2019 Du et al. 2021). In dynamic reconstruction model, flow deformation defining a time-dependent function the IR + R as a fush-forward of some reference function elo. ie., th d.. 1/n- A key advantage of a flow-based deformation model is that its generating velocity field often admits simple characterizations, facilitating the integration of priors into the model. For example, restricting di be volume-preserving can be achieved by imposing the constraint div(v) = 0 (Eisenberger et al. 2019).
be incorporated by
However, recovering v, values in the case the = dea s/o not explicit. Typically, this is achieved by solving the continuity equation?
Øt (x) div (v)(w)), (æ)) 0, Vx e IRd,
which necessitates a numerical simulation. This introduces computational challenges for training flow-based models, as errors in the numerical simulation can destabilize the optimization process. Therefore, to overcome this hurdle, our framework assumes a reconstruction model consisting of functions 4/ that are simulation-free i.c., each evaluation of th requires only a single step. However, since we advocate for deformation prior class formulation based on velocity fields, key challenge lies in controlling a simulation free v/ to adhere to such prior. Addressing this challenge is central aspect of our framework, as outlined in the following section.
Asssuiig obeys conservation law, where continuously deforms tht-
3

---

Published as a conference paper at ICLR 2025
3.2 FLOW REMATCHING
We assume that for time-dependent reconstruction function, 4/2 v,,, there exists an underlying flow di such that vi can described as a push-forward by Φt. We refer to Φi as the reconstruction flow and denote its generating velocity field by u. Under our assumption that by is simulation- free, neither 6e nor Uy is directly accessible. Nevertheless, for now, we assume access to an element i V, where V represents the of all the possible
wwwnsttrucctoongeerattiin velocity fields. This assumption will be relaxed later. Let P c {uel u: R R_ R be prior class of velocity fields to which or should belong. In Section 4 we discuss various choices for the prior class P.
In some of the choices for P. requiring o P could be over-restrictive, conflicting with the fact that Ut also adheres to generate the reconstruction flow. Hence, an appealing objective would be to optimize t2 that it is the closest element to P out of the set V. We suggest optimization procedure mimicking the alternating projections method (APM) (Deutsch) 1992). The APM is an iterative procedure where alternating orthogonal projections performed between two closed Hilbert sub-spaces V and P. Specifically, u projv (projr(v*)) guarantees the convergence of a to dist P). Following this concept, our next step find suitable notion for defining the projection operator for reconstruction generating velocity fields.
Since iy is unknown in our case, we suggest leveraging the continuity equation which provides both sufficient and a necessary condition for the generating velocity field of de in terms of 1/ and its partial derivatives. Specifically, we introduce projection procedure to recover the closest element in the prior class to the reconstruction flow by solving the following matching optimization problem:
ul u(.,t) 1) arg min p(u,, wh),
(5)
u,eP
where,
p(us, shx) = -f
1/e (2x) 1 Ot
8
+ div (vr(a))u dx.
(6)
This procedure is illustrated in the right inset, where it (red dot) is the closest point to i on P. Notably, neither oi nor i, appear in equation 5 aligning with our objective of relying solely ehc
Following the alternating projections concept, the matched ne should be projected back onto V propose better candidate for This corresponds to a flow matching problem in ug. We refer to this procedure ReMatching and introduce the flow ReMatching loss, LRM. matching loss striving for the reconstruction flow to match ur. That is,
P
LRM =E (/))
(7)
where u, is solution equation and denotes solely the parameters of vh.
The ReMatching loss is designed to supplement a reconstruc- tion loss TREC on the model parameters O. Thus, our frame- work's final loss for dynamic reconstruction training is
Algorithm ReMatching loss
Require: Solver for 5 times {t}}
LRM
for t E {tz} do
I(0)= = TREE (0) + ATRM
(8)
ux(+) + sollvollloooooso velveeep,. vh)-))
where > 0 is hyper-parameter. Note that in practice, the approximated by a sum using ran- dom samples {t, } CTO, 1]. In addition, for the ReMatching procedure to be seamlessly incorporated into reconstruction training process, it is essential that problem can be solved efficiently. To support this, in section 5 we provide linear constructions for P that are sufficiently expressive to encompass the consid- ered prior classes while allowing efficent solutions to equation 5 Algorithm [ summarises the details of computing 7 Note that calculating Vo does not necessarily require the cumbersome calculation of Ve arg min,, ((((,t),, vt). since according to Danskin's 's theorem (Madry 2017). Vop( u4 2/2) = V min,.ep p(u,, ).). Additional implementation details regarding the losses can be found in the Appendix.
LRM LRM + p(u()),)
integral in equation
end for Return: LRM
4

---

Published as a conference paper at ICLR 2025
4 FRAMEWORK INSTANCES
This section presents several instances of the ReMatching framework discussed in this work. One notable setting is when V = Raxd l.e, the (n ",,," )²² where each 7² IR, Rd. In this case, equation becomes:
2 p 1)) = Σ (1))))
99)
dt
Details this derivation can be found in section 8.1.2 For the settings where V C² (R"),, equation 6 involves the computation of spatial integral, which can be approximated by sampling set of points { Moreover, taking into account that all prior classes incorporated in this work are divergence- equation 6 becomes:
plut, 1/2x)
Σ
8 0t
v
(x.) (V))(
(10)
since div (1) (x))) (V. ur (x)) the (z) (x))
We now formulate several useful prior classes of velocity fields P. A key feature of all the following constructions is their reliance on linear parameterizations, capitalizing on the fact that linear subspaces are sufficiently expressive to represent the velocity-based prior classes considered. This approach enables the use of efficient solvers for problem 5 reducing the computational task solving a system of d linear equations, with run-time complexity of at most On))
4.1 PRIOR DESIGN
Directional restricted deformation. In certain scenarios, is safe to assume that the reconstruction flow can only deform along specific directions. For example, in an indoor scene, where furniture is placed on the floor, deformations would typically occur only in directions parallel to the floor plane.
Let span (v),-. "",, ")) 1 flow remains static. Then, the prior class becomes:
I s d and {",,-., y is predefined orthonormal basis in which the
P (u(r,, 1), vm.) 0, Vm [4]
(11)
When considering the matching minimization problem 5 in the settings of equation 9 we get:
min Σ (u
(12)
dt
dt
where V = [v1, r..)). For the settings involving equation 10
the matching minimization problem is
solved by:
11 0 min Σ
0 0
(1- (VV)(x;), V. VVVV V.VV((
w/ (V
Σ
(m;
U, 1=1
0t
(13)
where V, = (1-VVT).
Rigid deformation. One widely used prior in the dynamic reconstruction literature is rigidity, i.e., objects in a scene can only be deformed by rigid transformation consisting of a translation and an orthogonal transformation. In a simple case, where it is assumed that the underlying dynamics consists of one rigid motion, the reconstruction flow would be of the form
(t) R(t)x0++ b(t)
(14)
with R(t) e 0(3) and b(t) e R², Differentiating y and solving for æn yields that
d
dt 70 R(()R (1)((()))))))))))))))))))))))))) b()) + b(1).
(15)
matrix, Since R(t)R² (t) is a skew-symmetric matrix, we suggest the following natural parameterization for the prior class
Pnr (uelu(x, A, + b A, Rxdd A, -AY A,,,u b, Rd).
(16)
5

---

Published as a conference paper at ICLR 2025
Substituting Pı1 in problem
using equation Sarioon@iiilll yields the following minimization problem:
Ach + b, di mmil s.t. A. A
min (Ar,ba)
(17)
For the settings involving equation 10 the minimization problem becomes:
8
min (A.
Σ
th (α,)
(Vala A,z, bi)
s.t. A,= -
(18)
at
Importantly, both 17 and 18 are constrained least-squares problems. Thus, as detailed in Lemma they enjoy an analytic solution that can be computed efficiently.
Yolume-preserving deformation. So far have only covered prior classes that may be too simplistic for capturing complex real-world dynamics. To address this, a reasonable assumption would be include deformations that preserve the volume of any subset of the space. Notably, the rigid deformations prior class discussed earlier strictly falls within this class as well. Interestingly, flows are characterized by being generated via a divergence-fres velocity field, i.e., div u 0. To this end, we propose the following prior class:
Pirr = Amm=uuluua (x)) Σ 3= 3x1 R
(19)
where for each basis by R2d Rd. we assume that div(b,) 0. Clearly, di, 0 for any choice of e IR Taking into account that div curlu 0, we follow Eisenberger al. (2019). and incorporate the following basis functions:
by (x) {curl (0)(z)e() curl (d)(())))
(20)
where o, : R. (x)) n sin (jite( x with ji N denoting the frequency for the thh yields the following
coordinate of the jth basis function. Combining this prior with equation g minimization problem:
min Σ
Σ
B,bb((i)
(21)
i=1
Similarly, for the case of equation 10 we get:
Ot 2 (zx) + (Vii(a))) Σ 3,00 3,0,
min
(22)
In particular, both minimization problems 21 and 22 correspond standard least-squares problem and have an analytic solution that can be efficiently computed. A key decision involved in using the prior class PI is select the number of basis functions k. However, setting & equal large value would make PIL overly permissive, effectively neutralizing the Re.atching loss. To address this, in what follows, we propose an additional procedure for constructing more complex prior classes, based on an adaptive choice of complexity level.
Adaptive-combinatie of prior classes. To address the challenge of set- ting the complexity level the prior class, we introduce an adaptive (learn able) construction scheme for prior class. Let wy (æ,t) Rd x IR, + IR, 1 S S k, be learnable functions, which are part of the reconstruction model, ie., wy (t) E W and wy are normalized, i.e., Σ (x, = 1. The de- tails of wy architecture are left to Section We can construct complex prior class by assigning simpler prior classes to different parts of the space, according to the weights wy. For example, let consider piece-wise rigid deformation prior class defined as:
Figure 1: A vector field in Pv.
Piv uolu( uolu(xt
Σ wy (,, e Pnr for isjsk 1=1
(23)

---

Published as a conference paper at ICLR 2025
In similar manner, can also combine Py with rigid deformations and derive prior class defined as:
Pv (x) - wy
E P, E Prr for 25 k)
(24)
Figure 1 iillustrates element in Pv. with weights wi dividing the plane to restricted up direction deformation above the diagonal, and a rigid deformation below the diagonal. Note that directly sub- stituting cnaptive-commbnattion prior class in S would no longer yield linear problem. Therefore, we propose to use a linear problem that upper bounds the matching optimization problem of For example, in the case of equation with Piv. we can solve:
min ((A)),;;)) =1j=1
ΣΣ
wi (T, Asci + by dt
s.t.
-A
(25)
Using Jensen's s inequality, it can be seen that 25 upper bounds the matching optimization from S
Now, problem 25 can be solved efficiently, as it corresponds to a weighted least squares problem that solvable in parallel for each [A:], similarly to problem 17
Lastly, incorporating Pı1 into equation | results in a non-standard least-squares problem with a constraint. The following lemma, with its proof provided in the Appendix, formulates the solutions for using Piv in equation 5 covering problems 17 and T8 as a special case.
Lemma 1. For the prior class Piv, the solutions given by,
(Ajt. bys) to the minimization problem 25
vech(AAt) bjs
}
vec W W. Je 5,iit
P77
1 1 W, d 1r') |
where F [_d.--.-0']"" Rnnd r.
de 2
RRdd W5r = Ei Ei ())
with {e,} as the standard basis in R", (Ac R anfi-symmetric matrix Ajt, and the matrix ps depends solely on Je Tr, and W
denotes the half-vectorization of the
The solutions (Aju, byt) ) to the minimization problem with 10 are given by.
[
vech(AAe)
] P Σn (, (x -g/x
bjt
G² W j
where gi [Vab(<:))]", G. gg. ".. "r
E Rxddd
8, 8 (x:).
[s] [ "7 ER....
IMPLEMENTATION DETAILS
In this section, we provide additional details about the dynamic image model W employed in this work, based on Gaussian Splatting (Kerbl et al. 2023). We provide an overview of this image model, followed by details about the dynamic model used in the experiments.
Gaussian Splatting image model. The Gaussian Splatting image model is parameterized by a collection of n 3D Gaussians augmented with color and opacity parameters. That is, 0 a
{u', E', c',a"' ) 5-1 with 4² e R33 denoting the 7th Gaussian mean, Σ' e 133x3 its covariance ma- trix, c' e 3 its color, and a² e RR its opacity. To render an image, the 3D Gaussians are projected to the image plane to form collection of 2D Gaussians parameterized by [Hâp, Ein) Given K, E denoting the intrinsic and extrinsic camera transformations, the image plane Gaussians parameters are calculated using the point rendering formula:
Ep²
MDD K (Exc)z E J"
(26)
where J denotes the Jacobian of the affine transformation of 26 obtained by siphadblending the ordered by depth visible Gaussians:
Lastly, image pixel I(p) is
I(p) dd'' (p) T (1-1a a'- (p)).
i-1
(27)
1=1
where o² (pp) -epp(= (p (p- ) "(EE>) (p (2n)
7

---

Published as a conference paper at ICLR 2025
Dynamic image model. model as:
We utilize the Gaussian Splatting image model construct dynamic
V. = (1'
a², (us((t))
(28)
where (1) = fu(u', t). (t) fx((,, i), (t)(t) = ef (1, t)).
We R
follow Yang al. (2023) and each of the functions: Snx x x - IR, /x RD fw R F x R- Rx is a Multilayer perceptron (MLP). For more details regarding the MLP architectures, we refer the reader to the Appendix. Note that the model element way (t) is only relevant to instances where the ceraptive-combinatee
prior class is assumed. Lastly, in our experiments we apply the ReMatching loss for μi' + μe' (t), and for time-dependent rendered images 4,
Training details. We follow the training protocol of Yang et al. 2023). using n 100K 3D Gaussians. Training is done for 40K iterations, where for the first 3K iterations, only (w', Σ " are optimized. In instances where the adaptive-combl combination prior class is applied, we supplement the ReMatching optimization objective with an entropy loss on the weights Wij as follows:
We initialize the model
k -
1
" wij F1
Lentropy
wiy log
(29)
j=1
Lastly, for all the experiments considered in this work, set the ReMatching loss weight À 0.001.
EXPERIMENTS
We evaluate the ReMatching framework benchmarks involving synthetic and real-world video cap- tures of deforming scenes. For quantitative analysis in both cases, we report the PSNR, SSIM (Wang ct al. 2004)and LPIPS (Zhang et al. 2018) metries. Additional evaluations, including experiments on more synthetic and real-world datasets, hyperparameter ablation, and the framework's 's applicability to an alternative dynamic image model, are provided in the Appendix.
D-NeRF synthetic. D-NeRF dataset (Pumarola et al. 2021 comprises of 8 scenes, each consisting from 100 to 200 frames, hence providing a dense multi-vies coverage of the scene. We follow D-NeRF's evaluation protocol and use the same train/validation/tess split at 800 x 800 image reso- lution with a black background. In terms of baseline methods, we consider recent state-of-the dynamic models, including Deformable 3D Gaussians (D3G) (Yang al., 2023). 3D Geometry-awan Deformable Gaussians (GA3D) (Lu et al. (022)). Neural Parametric Gaussians (NPG) (Das et al.) 2024). and K-Planes (Fridovich-Keil et al. (023). Note that some of these baselines incorporate prior regularization losses such as local rigidity and smoothness to their optimization procedure. Table summariizs the average image quality results for unseen frames in each scene. We include two variants of our framework: i) Using the divergence-free prior Pizri ; and ii) Using the adaptive- combination prior class PIV or the class Pv specifically for scenes that include floor component. Figure provides a qualitative comparison of rendered test frames, highlighting the improvements of our approach, which: i) produces plausible reconstructions that avoid unrealistic distortions, e.g.. the human fingers in the jumping jacks scene; ii) reduces rendering artifacts of extraneous parts, especially in moving parts such as the leg in the T-Rex scene.
Bouncing Balls
Hell Warrior
Book
Jompinglacks
Method
LPPISS
PSNR1
SSIM
1PPPS 1
PSNR
SSIM!
LPPPS
PSNR
SSIM 1
LPIPS
PSNK
SSIM
K-Plaee
0,0242
37.75
0.9929
0.1074
U.9316
3U3655
29.46
U.0417
31.73
0,9715
NPG
0.0537
0.0450
33.29
0.9735
0,0345
33.97
0.9628
GA30
0,0093
40.26
0.9950
0.0100
41.30
0.9871
00124
37.28
0.9887
0.011
37.00
0.9687
D3G
0,0089
41.52
0.9978
0.0261
41.26
0.9928
0.0155
37.00
0.9905
0.0137
37.59
0.9930
Ours
0:0087
41.84
0.9979
0.0244
41.59
0.9933
0.0661
37.19
0.9909
0.0144
37.72
0.9911
Lego
Matant
Stund Up
T-Rex
Method
I.PIPS
PSNR
SSIM
1PPPS 4
PSNR
SSIM
LPIPS
PSKR
55IM 1
I.MPS
K Planes Sooe
0,0472
25:15
0.9431
0.0215
35.30
0.9825
3U0211
36.55
0.9831
0.0284
30.41
0.9778
NPG
0,0715
24.63
0.9312
0.0311
35.02
0.9840
0.0257
38.20
0.9889
0,0310
32,10
0.9959
GA30
0,0446
24.87
0.9220
0.0050
42.39
0.9951
00082
43.56
0.9968
0.0100
37,70
0,9929
24.91
0.0105
0,9056
Oues Ous (Pay or Pr)
6,0503
24.80
0.9523
0.0067
0.9966
0.0885
43.09
0.9969
0.0105
38.07
0.9058
0,0456
24.95
0U5537
0.0065
42:40
119968
300051
44:31
0.9971
0.0003
38.38
0.9961
Table 1: Image quality evaluation on unseen frames for the D-NeRF dataset (Pumarola et al.
2021
8

---

Published as a conference paper at ICLR 2025
GT
GA3D D3G
Ours
GT
GA3D
D3G
Ours
Figure 2: Qualitative comparison of baselines and model on the D-NeRF dataset (Pumarola et al. 202I). We note that our framework consistently produces high fidelity reconstructions, accurately capturing fine-grained details, as highlighted in the blue boxes.
HyperNeRF real-world. The HyperNeRR dataset (Park et al. 2021b) consists of real-world videos capturing a diverse set of human activities involving interactions with common objects. We follow the evaluation protocol provided with the dataset, and use the same train/test split. In table report image quality results for unseen frames on scenes from the dataset: Slice Ba- nana, Chicken, Lemon, Torch, and Split Cookie. Figure 3 shows qualitative comparison to the baseline D3G (Yang et al., 2023).. Our approach demonstrates similar types of improvements as noticed in the synthetic case providing more real- istic reconstructions, especially in areas involving deforming parts.
Scene
LPIPS
PSNR
SSIM
GA3D
0.4160
25,34
0.6722
D3G
0,3692
24.87
0.7935
Slice Ranana
(P1zz)
25.08
0.7992
Ours (Prv
0.3673
25.28
0.8025
GA3D
0.4721
25.13
0.7556
D3G
0.3030
26.66
0.8813
Ours (Prr)
0,2987
26.74
0.8836
Ours (Pyv)
0.3044
26.80
0.8835
GA3D
0,3252
28.37
0.7596
Lemon
Ours (Pzzz)
0,2760
27.91
0.8842
Ours (Pyv)
0.2675
28.30
0,8883
GA3D
0.3278
23.79
0.8174
D3G
0,2340
25.41
0.9207
ReMatching dime-dependent image. In this experiment we validate the applicability of the ReMatching loss for controlling model solutions via rendered images. To that end, we apply our framework with the PIII prior class the Jump- ing Jacks scene from D-NeRF on single specific front view through time. The qualitative compar- ison to D3G (Yang et al.) 2023), shown in the Appendix, supports the benefits of prior integration in this case as well, demonstrating more plausible reconstructions in areas involving moving parts.
Torch
Ours (Pizr)
0,222
26.00
0.9251
Ours (Pyv)
0,2260
25.62
0.9229
GA3D
0.9290
D3G
0.0971
32.61
0.9657
Split Cookie
Ours (Piz1)
0.1097
31.31
0.9600
Ours (Piv)
0.0937
32.67
0.9667
Table 2: Unseen frames evaluation for the Hyper- NeRF dataset (Park et al. 2021b).
Adaptive-combination prior class. Employing the adaptive- combination prior classes PIv and Py with learnable parts assignments {wyy} raises the question of whether the learning process successfully produced assignments { [w;j} that align with the scene segmentation based on its deforming parts. Fig- shows our results for test frames from the Bouncing-Balle and Lego synthetic scenes (left), and the Chicken real-world scene (right). For comparison, we include the results of the Segment Anything Model (SAM) (Kirillov et al. (023), which
SAM
Ours
SAM
Onns
Figure 4: Part assignments for the adaptive-combination prior class. mostly influenced by color variations. Consequently, SAM often over-segments the scene or incorrectly merges independently moving parts. See the supplementary material for additional segmentation results.

---

Published as a conference paper at ICLR 2025
GT
D3G
Ours
GT
D3G
Ours
Figure Qualitative comparison of our method D3G YYang et al. 2023) the HyperNeRF dataset (Park et al. 2021b). Our framework yields more accurate reconstructions, in particular around moving parts.
CONCLUSIONS
We presented the ReMatching framework for integrating prior deformation classes into dynamic reconstruction models. In addition to offering useful constructions for velocity-field-ased prior classes, a key focus of this work was the development of the ReMatching loss. This loss function optimizes for solutions that remain close as possible to the desired prior class, rather than strictly enforcing membership, thereby minimizing the tradeoff between fidelity and the ad advantages of using priors. Our experimental results validate this approach, showing that ReMatching solutions successfully adhere to the desired prior while achieving high-fidelity reconstruction. We believe that the generality with which the framework was formulated will enable broader applicability and that future ad advancements in this framework could be relevant to wide range of dynamic reconstruction models. An interesting avenue for future research is the development of velocity selocity-field-ii based prior classes emerging from video generative models, potentially using ReMatching formulation for time-dependent image pixels color. Another promising direction is the design of richer prior classes to handle more complex physical phenomena, such including liquids and gases.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank Jonathan Lorraine and Heli Ben-Hamu for their insightful discussions and valuable comments.
REFERENCES
Michael S. Albergo, Nicholas M. Boffi, and Eric Vanden-Bijnden. Stochastic interpolants: A unifying framework for flows and diffusions, 2023. /tt///tps///axx....oo//////222.0
Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Aditya Grover, Maximilian Nickel, Ricky TQ Chen, and Yaron Lipman. Matching normalizing flows and probability paths on manifolds. arXiv preprint arXiv:2207.04711 2022.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 130-141, 2023.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Mengyu Chu, Lingjie Liu, Quan Zheng, Erik Franz, Hans-Peter Seidel, Christian Theobalt, and Rhaleb Zayer. Physies informed neural fields for smoke reconstruction with sparse data. ACM Trans. Graph., 2022.
Devikalyan Das, Christopher Wewer, Raza Yunus, Eddy IIg, and Jan Eric Lenssen. Neural paramet- ric gaussians for monocular non-rigid object reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10715-10725, 2024. 2024.
Frank Deutsch. The method of alternating orthogonal projections. In Approximation theory, spline functions and applications, pp. 105-121. Springer, 1992.
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14304-14314, IEEE Computer Society, 2021.
Marvin Eisenberger, Zorah Lähner, and Daniel Cremers. Divergence-free shape correspondence by deformation. In Computer Graphics Forum, volume 38, pp. 1-12. Wiley Online Library, 2019.
Sara Fridovich-Kaa Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 12479- 12488. IEEE, 2023.
Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. CVPR, 2023.
Haoyu Guo, Sida Peng, Yunzhi Yan, Linzhan Mou, Yujun Shen, Hujun Bao, and Xiaowei Zhou. Compact neural volumetric video representations with dynamic codebooks. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
Bernhard Kerbl. Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL Bittps://repoo sam, inria. fr/ fungraph/ http:phhhhhgguuuuugoe
sp att.ing/
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.0264/ 2023.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis, Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEF, 2024.
11

---

Published as a conference paper at ICLR 2025
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, arXiv preprint arXiv:2110.02777, 2022.
Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhöfer, Yaser Sheikh, and Jason M. Saragih. Mixture of volumetric primitives for efficient neural rendering. ACM Trans. Graph., 40 (4):59:1-59:13) 2021.
Zhicheng Lu, Xiang Guo, Le Hui, Tianrui Chen, Min Yang, Xiao Tang, Feng Zhu, and Yuchao Dai. 3d geometry-aware deformable gaussian splatting for dynamic view synthesis. arXiv preprint 00XY::2404.06270, 2024.
Aleksander Madry. Towards deep learning models resistant to adversarial attacks. arXiv preprint wrXiv:/706.06083, 2017.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5379-533799 2019.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Scitz, and Ricardo Brraan-Br Nerfies: Deformable neural radiance fields. 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Ricardo Martin-Brualie and Steven M. Seitz. Hypernerf: higher-dimensional representation for topologically varying neural radiance fields. CoRR, abs/2106. 13228, 2021b. URL https
//arxiv. http://..orgg/b////60/ 111/221111.2211888
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19318-0027,, 2021.
Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojeic, Srinath Sridhar, and Leonidas J. Guibas. Caspr: Learning canonical spatiotemporal point cloud representations. In Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Zhongzheng Ren, Xiaoming Zhao, and Alex Schwing. Class-agnostic reconstruction of dynamic objects from videos. Advances in Neural Information Processing Systems, 34:509-522, 2021.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. International conference on machine learning, pp. 1530-1538. PMLR, 2015.
Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 2955)2732--2742 2023.
Olga Sorkine and Mare Alexa. As-rigid-as-possible surface modeling. In Symposium on Geometry processing, volume 4, pp. 109-116. Citeseer, 2007.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada. October 10-17, 2021, 2021.
Tuan-Anh Vu, Duc Thanh Nguyen, Binh-Son Hua, Quang-Hieu Pham, and Sai-Kit Yeung. Rfnet-4d: Joint object reconstruction and flow estimation from 4d point clouds. Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 - 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXIII, 2022.
12

---

Published as a conference paper at ICLR 2025
Chaoyang Wang. Peiye Zhuang, Aliaksandr Siarohin, Junli Cao, Guocheng Qian, Hsin- Ying Lee, and Sergey Tulyakov. Diffusion priors for dynamic view synthesis from monocular videos. CoRR, 2024.
Hsin-Ying
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309. 13101, 2023.
Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. CoRR, abs/2004.01294
2020. URL URLhttpss: aawiiv.ivvv.. org/ abs/////////// 20204.01222
Hong- Xing Yu, Yang Zheng, Yuan Gao, Yitong Deng, Bo Zhu, and Jiajun Wu. Inferring hybrid neural fluid fields from videos. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Raza Yunus, Jan Eric Lenssen, Michael Niemeyer, Yiyi Liao, Christian Rupprecht, Christian Theobalt, Gerard Pons-Moll, Jia-Bin Huang, Vladislav Golyanik, and Eddy 1lg. Recent trends 3d reconstruction of general non-rigid scenes. In Computer Graphics Forum, pp. e15062. Wiley Online Library, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
APPENDIX
8.1 PROOFS
8.1.1 PROOF OF LEMMA
Proof. (Lemma
Let rú, T., W5 be given. Without loss of generality, we show the proof only for individuals j [k]] and t. So in order to ease the notation, in what follows omit the subscripts t and j. Let A E 12 dddd and b Rd, and define [wo,...., = W1. First, we show that Σ. ne A b b- 10ll can be reformulated as weighted norm-squared minimization problem in A and b. That is,
Σ Wi A + b 7-112 Σ UrA wSy w
T AT-
(30)
21r w - b) W AT tr/ -b) b) -b) -6)
(31)
tr AT WTAT 2tr(T 1-- I
(32)
( 16") (
(33)
W -a--"(((116)))
(34)
Next, we consider the following optimization problem:
min A
W - (f---A"----116111 s.t. A - A
(35)
13

---

Published as a conference paper at ICLR 2025
Use the fact that A -A to define the following Lagrangian,
((A.b.A) VW 16" +
+ 1-"--11"(++ +1]l (A++))+
(36)
Then,
0A = 21 W(TA-10'+T)) +
0C Thus, 8A yields that - W 16" +f) symmetric. Then, using again the fact that A = - A", we get that,
I' WTA+ AT WT + 61 - W16
W W (37)
Now, taking the derivative w.r.t. gives,
dC
0b = -21 W(TA-16'+ +1)
and, 8C bb 0, yields that,
① TA + b" = TT
(38)
where
Vectorizing the L.HS of 37 gives,
TWI
WT+ a 1.) Davech + w
(39)
where Da is the duplication matrix transforming vech(A) to vec(A), with vech (A) denoting the half-vectorization the anti-symmetric matrix A. Similarly, vectorizing the LHS of 38 yields,
1''1 (1a @ 1" WT) Davech
(40)
Based on 39 and 40 we can define the following block matrix:
QQQ'Da R -=r W1 @ Ia - Ia W1
p
(41)
S
T=
where Q' - WT WT' Ia, and, S' -- (Ia 1 WT). Then, let,
U- RT 1)- La (Q'- R- -Q'-
(42)
where La the matrix satisfying Dalad = Ig.. Consequently,
U
-UR
p-1 =
(43)
S' (Q' RRS'
Ta+ S' (Q' RRS') R
and,
vech(A)
vec WT -
p-1
(44)
b
TT r
Now, for the second part of the lemma. Let g: = [Vrh( ,8 = 21 (ax:). Consider the following energy.
L =
w. (g (Ax; + b) s.)²
(45)
Note that,
g Ax, = y.a
(46)
where a: vec(A), and y: æ; gi. Then.
n L
Σ 1
wi (a youk a + be g.gt b+ 2a y. b + 2 s. 28,a" y: 2?)
(47)
14

---

Published as a conference paper at ICLR 2025
Define the Lagrangian,
((a,b,)) - a" Eywwy a+b G" Σ Ly..ggt b+
(48)
2s WGb+ 2a T Luiyis + +t Wt+ (a Pa)
(49)
where the permutation matrix s.t. vec(A") Pa.
Then,
ac
da =22 =2 w.yy a 2 w.y.g b+ 2 w.yss + + PA
(50)
Equating the above to 0 and unvectorizing it, yields the following matrix equation,
Σ 2w, 9.9 bat s.gaz))= (A+
(51)
yielding that the LHS is symmetric matrix. Therefore,
w. wg Ar,a a.g ba? sggaa Σ wa A g: ar 9:g s.a.g.).
(52)
Rearranging the above and half-vectorizing both sides yields that,
Σ wi (a. gigt 9:g @ a... Davech(A)) + ww ww @ g.g g:g ax)b=
(53)
( g.2l))
(54)
Now,
Oc ab
0
(55)
yields that,
GT Σ = Wss.
(56)
Therefore,
P Q == Q'Da R= (1 g. 9:9, x;)
(57)
S'Da
where S' = Ei w.g.y, and Q' = En w (x @ g.g g.g. @ )z,e)) Then, let,
= (Q RT S) La (Q' RT sy)
(58)
where La the matrix that satisfies DaLa Ig. Consequently,
-RTT p- = -T-sS - RT-1S') T S') RT-1
U
(59)
8.1.2 CONTINUITY EQUATION CONSTRAINT DERIVATION FOR Raxd
In the main text, we stated that in the case when V Rudd i.e., 1/2 = ("))" equation becomes:
p(u the = Σ (u(t))
(60)
dl
To this formally, let 6(x a) denote the Dirac delta generalized function concentrated around satisfying
8(2 a) Var a,
(61)
15

---

Published as a conference paper at ICLR 2025
and,
f (x))(x= =()),
(62)
for any test function Φ. Consider the (xx) = Σn- (1). where (x) 8(x - -1)) Note that under this definition of Vr, V is in fact the space of generalized functions. Then,
2 (vō(z -;),
(63)
and, using the chain rule as applied in the simplification of equation 10 we have that,
div vf((()) (vō(* - 7:), u u(x)) 6(x - 7)) div(u(x)))
(64)
Substituting these computations in the continuity equation 4 yields that,
0 ==
w/ (x) + div (ar(a)) (x)) dx 2
(65)
0t /
ot (1((()) + div (1)(( (z)) da
(66)
di Is (vol0- 12), (6)(x-1))--0 -1)))-11)) -xxi)= + (vō(= 1alaa-nii ()).((x)) 8(x 7) div( un (x) dz
(67)
/ʃ (v6(0- x)) dt
d dxd dx+ f 8(2 ( div(u(z)) dx
(68)
Now, under the assumption that div(u)= 0 almost everywhere, using 62 yields that the second term in the last equation vanishes. Therefore,
| vō(x (5( - 71), u(x) |s 8(x 78) ( log 6(x 71), u(x)
d dx
0-
dt 11
(69)
d
dt 7t 1
dx
2
(70)
f 6(x >/) | log 6(x - 7i) ee(z)
d d.
Yt dx, dt
(71)
where we applied the Cauchy- inequality in the final step. Therefore, d
f 6(zz 7 || log ((x ~1) |u(x))
de dx = O.
?
(72)
Applying property 62 yields that equation 72 can be true only if when æ 2h, we have that,
d
=0.
(73)
dt
Utilizing this constraint for each i, derive equation
9
8.2
ADDITIONAL IMPLEMENTATION DETAILS
8.2.1 ARCHITECTURE
We first describe the construction of the Gaussian Splatting dynamic image model referenced in section 5 An illustration of this model is presented in Figure Figure the applied losses. The time invariant base of the model is optimized throughout training and consists of the following set of parameters 0 = (nr St -''c',a i Ji-1 with Gaussian mean fa¹ R scaling S IR3, rotation quaternion R' e RR4 color R and opacity a' IR. The covariance matrix Σ is calculated during the rendering process from the temporally augmented scaling and rotation parameters.
The time dependent deformation model transforms the time invariant Gaussian mean μ and selected time into the deformation of the mean, scaling, rotation and the model element w in the case of the addptive-combination prior.
We generate positional embeddings (Mildenhal et al. 2020) of the time and mean inputs, which we pass to the deformation model Multilayer perceptrons.
Emb (1) R. T
16

---

Published as a conference paper at ICLR 2025
3+3+3
d:
3+3+3++
H W
n
Referemoe Conzziaza Time
Time varying point. lentures
Time varying Ganeeians
Rendered Image
Figure 5: Illustration of the architecture for Ve used in the experiments, based on (Yang et al. 2023) Reference Gaussians parameters are propagated to time through shared function, vtt implemented as an MLP with positional encoding features to compute time-varying point features of dimension de These features then processed by second shared function, v. to generate time-var varying Gaussians parameters. Finally, given a chosen viewing direction, the Gaussian Splatting rendering model is used to produce rendered image.
LRec
LRM
LRM
Rendered Image
Input Image
Rendered Image Time-varying Ganssians positions
Figure 6: Illustration of the losses applied for learning V. The reconstruction loss, Lree is applied on a rendered image to recover a corresponding given image of the scene. The ReMatching loss, LRM- can be applied either rendered image time-varying Gaussians positions.
Embmean (1¹²) )) : TR IR rgemaaaen
The deformation model is made up of layers of the form:
(n,, din, dow) X - v (XW+XX +16²)
where v Softplus with 3 100.
For the deformation of the mean, scaling and rotation, the model takes the same form with minor differences in the final layer depending on the deforming parameter.
Emb (t) (/, drime emb. 256) (n, 256, d+)
[T, Embmean (11) - 1b( dr dmean emb; 256) - (n, 256, 256) - (n, 256, 256) (n1 256 256) [T, Embmean (1)) (n, n, 256, 256)] + (ne, d, + dmsan emb 256, 256) (/, 256, 256) (n, 256, 256)
Mean (n,, (2,2 jut)
Scaling: w - (n, 256,3) 3) S(t)
Rotation (n, 256, 4) R(t)
For the prediction of the w we use a shallower Multilayer perceptron.
[T, Embmean (a(t),, Emb (14) + (n, dy 2 drea emb, 256)
(n,, 256, K) Softmax w(()
17

---

Published as a conference paper at ICLR 2025
8.2.2 HYPER- PPPER-PRRMMEERS AND TRAINING DETAILS
We set dmmaa emh 63, drime = 13 and d, 30. For optimization we use an Adam optimizer with different learning rates for the network components, keeping the hyper-parameters of the baseline model (Yang et al., 2023).
In the case of the ddlapive-combinattion prior we select based on a hyper-parameter search between 1 and 35. The optimal value for most scenes ranges between 5 and 15, though the number also depends on the selected composition of priors. For example, single volume-press pressevving class can supervise multiple moving objects as opposed to a single rigid deformation class. We use the ReMatching loss weight À 0.001. When supplementing the ReMatching loss with an additional entropy loss, we use 0.0001 as the entropy loss weight.
In calculating the partial derivatives of 4/+ we note that the input dimension for predicting the is relatively small - for time or d for spatial coordinates compared to the output dimension, which can be n for spatial ReMatching or automatic differentiation proves to be more efficient than backward- specific computation, both computationally and in terms of memory usage. Consequently, we utilize forward- mode autodiff to compute the partial derivatives of th, required for the ReMatching loss. Once the ReMatching loss is incorporated, we employ backward-mode autodiff to compute the gradient of the overall loss with respect to the model parameters.
in image-space ReMatching, Given this, forward-mode
differentiation for this
8.2.3 REMATCHING RENDERED IMAGE
The reconstruction model architecture in the case of the image ReMatching is the same as for the other experiments. At initialization we select fixed viewpoint for the evaluation of the image space loss, which is kept throughout training. At every iteration we sample a random time and evaluate the ReMatching loss from the fixed viewpoint.
we calculate sample by choosing points that their image value is close to 0 after applying the following transformation:
For approximating equation 10
f(x) = -0.1
-0.1-Inn( In(1 - (ar) [at)- sign(x)
(74)
on the image.
Next, we compute the image gradient using automatic diffcrentiation and use our single class div-free solver to reconstruct the flow and calculate the loss.
8.3
ADDITIONAL. EVALUATION
8.3.1
3D GEOMETRY-AWARE DEFORMABLE GAUSSIANS
We provide additional set of experiments that show the versatility of our framework. Our evalua- tions of the framework in tables and were made by applying our framework to a reconstruction model inspired by Deformable 3D Gaussians YYagg al. 2023). In this evaluation, apply the same framework on a baseline following the 3D Geometry Deformable Gaussians (Lu et al.) 2024). Table shows our results compared to the baseline GA3D.
8.3.2
DYNAMIC SCENES DATASET
To further evaluate the efficacy of the ReMatching framework in practical applications, we consider the Dynamic Scenes dataset (Yoon et al.) 2020), which captures forward-facing views of real-world scenes exhibiting complex dynamics. To that end, we selected 4 scenes from the Human, Interaction, and Vehicle categories, consisting of monocular videos with approximately 80-180 frames for training and an additional 20 frames reserved for testing. (jsowss qualitative comparison to the D3G Yang et al.) 2023) baseline, highlighting similar patterns of improvement as observed in earlier experiments. Specifically, our approach better preserves fine details, such as the truck's front lights (Truck) and the bottom teeth (Dynamic Face). Additionally, it demonstrates reduction in reconstruction motion artifacts, as evident in the humans in motion (Jumping) and the legs and head of the dinosaur (Balloon). Table presents quantitative evaluation, comparing two variants from
18

---

Published as a conference paper at ICLR 2025
Figure 7: Qualitative comparison of the ReMatching loss applied in the image space. Each group of 3 is showing Ground-Truth (left), Ours (center), and D3G (right).
prior classes, Pısr and Prv to D3G. These results correlate with the qualitative improvements discussed above.
8.4 RECONSTRUCTION FLOW EVALUATION
In this section, evaluate the ability of the ReMatching framework to recover the underlying reconstruction flow Φr. Since the reconstruction flow is generally unknown, we use the following simple flow generate training data:
Φ x:) R(t)
(75)
where R² ((t) = Ia- We evaluate our framework its two settings: i) equation 9 corresponding to V = Raxd, and, ii) equation 10. corresponding to V C² (IR.).
The V R case. We evaluate these settings for d 3, following a similar approach to the dynamic image model based on Gaussian Splatting described in Section 5 To construct the training set, we use reference scene of a colored box and apply the ground-truth flow dr, create dynamic scene consisting of multi-view captures over 12 time stamps. For the flow Φr, we set
Bouncing
Jumpinglacks
Method
LPIPS
PSNK
SSIM
LPIPS
PSNR1 1
SSIM
LPIPS 1 PSNR
SSIM
LPIPS
PSNK
SSIM
GA3D
B.003
43.42
0.9960
0.0175
3200
0.9827
0.0076
36,88
0.9155
QU0092
37.83
0.9924
0,061
00165
32.15
0.093
0.9927
Ours (Piy or P
0.0058
43.62
0.962
00163
3221
0.0077
37.01
09916
00078
0.993]
Lego
Mutuni
Stund Up
T-Rex
Method
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
Ours (1 1221
25.53
019099
0U0029
41.41
0.9M2
0.0028
42.24
019967
00052
39.12
0.9452
Durs (P Pe)
6,0300
25.50
0.9487
01028
41.55
8.9970
0J0027
43.24
0.9988
0.0649
39.18
0.9953
Table 3: Image quality evaluation on unseen frames for the D-NeRF dataset with our framework
applied on top of the 3D Geometry-awart Deformable Gaussians (Lu et al.) 2024). Image resolution scaled down to 400x400 pixels and the background is white, maintaining the settings of the GA3D paper evaluation.
19

---

Published as a conference paper at ICLR 2025
Balloon
Truck
Jumping
Dynamic Face
Method
LPIP
PSNK
SSIM
LPIPS
PSNK
PSNK
PSNR
Ours Foo
0.1992
26.96
0.9488
0.2782
25.49
0.9021
0.2730
22.89
0,8971
0.0794
29 30
0.976
Ours Pay
0.1578
20,95
0,2533
20.66
0.2501
25.83
05007
0.0793
Table 4: Unseen frames evaluation for the dynamic scenes dataset (Yagg et al. (023).
GT
D3G
Ours
Figure 8: Qualitative comparison of our method to D3G Yangg et 2023 on the dynamic scenes dataset Yoon et al., 2020).
Figure Training frames from the rotating colored box scene.
cos 2Tr
sin 2Tl
07
R(I)
sin 2nt
cos 2nt
Figure shows selected images from the training set. Two training
O
procedures considered: (i) baseline approach using only the reconstruction loss, similar the
20

---

Published as a conference paper at ICLR 2025
GT
Ours
D3G
Figure 10: Evaluation of unseen timestamps in the rotating colored box scene.
GT
Ours
D3G
Figure 11: Visualization of {1l (()))(())))))))))))))))))))))() from multiple timestamps.
D3G model; and, (ii) approach where both the reconstruction loss and the ReMatching loss optimized. For the ReMatching loss, we employ the global rigid motion prior class Prz. Figure TO compares ground-truth images from time-stamps unseen during training with the model's predicted renderings. Notably, the ReMatching loss allows the model to generalize in alignment with the ground-truth flow Φi. This is result of the ReMatching objective's ability converge matched priors un that accurately recover the ground truth velocities dt. To further support this claim, Figure
21

---

Published as a conference paper at ICLR 2025
Figure 12: Visualization of the training set made up of three functions (tr((;;))
illustrates the velocities of the dynamic Gaussian centers, {t)( These results demonstrate that the ReMatching loss effectively controls {i) resulting with solutions {id (1) that match the prior class Pıı and recover the ground-truth velocities 0 ot.
The V = (IR)) case. We evaluate these settings for d = 2. Using the flow described above, we define the following ground-truth scalar function, vGT R² + R, as:
(,,( =1 (S x) |... -
(76)
The training data is constructed using three time-stamps, specifically (0.0,0.25.0.55) The cos ाट sin ot parameter choices for this procedure are: b 0.2, and S diag (0.6, 1.4). R(t) - at cos
Figure 12 Visualizes the three distinct functions vor((, () that constitute the training set. To model th., we employ a multi-layer perceptron (MLP) architecture, described in Section 8.2.1 with the only modification of scalar output dimension in the final layer. For the reconstruction loss, we adopt the standard I, loss:
3 LRFC Σ E (1b (.) vor((x44))
(77)
For the ReMatching loss, we employ the global rigid motion prior class Ptr. Two training pro- cedures are considered: (1) a baseline approach where only the reconstruction loss is used as the optimization objective, and (2) our suggested approach where both the reconstruction loss and the ReMatching loss are optimized. Figure 13 displays the results of the trained models for times t e (0, 0.0665.00000 O. 125, 0.1875, 0.25, 0.31 0.375, 0.4375). Among these, only /. E {0, 0.25, 0.5), shown in the leftmost column, correspond to the training set frames. While both the baseline and our approach perform similarly the training frames, the results for unseen frames clearly demonstrate benefits of incorporating the ReMatching loss. Specificallyy, the ReMatching loss allows the model to recover the ground-truth flow di, avoiding the unrealistic distortions observed in the baseline results. To further illustrate this, Figure| depieicss the matched priors ut (white arrows) obtained by solving equation 5 alongside the velocity field of the ground-trui -ruth flow 3t or (green arrows). These comparisons show that the ReMatching loss successfully converges to matched priors u, that closely approximate the ground truth. contrast, the matched priors u, obtained with the baseline approach (without the ReMatching loss) deviate significantly from the ground truth. This emphasizes the importance of the reprojection procedure. Not all velocity fields in the prior class are suitable for guiding the optimization process, but the ReMatching loss ensures convergence an appropriate prior, enabling an accurate recovery of the underlying flow.
2
8.5 RUNTIME AND CONVERGENCE ANLAYSIS
We note that framework is applied solely during the training phase of the algorithm, leaving inference times unaffected. To evaluate computational efficiency, we measured the average time (seconds) for forward and backward pass over 100 iterations for varying sizes n of Gaussians sets.
22

---

Published as a conference paper at ICLR 2025
Ours
Baseline
Ours
Baseline
Figure 13:
Comparisons of sh converged solutions between the baseline and our approach, displayed order of increasing time (left to right, top to bottom).
Figure 15 presents the results, comparing the computation time of the ReMatching framework to the D3G baseline. The runtime analysis was conducted single NVIDIA RTX A6000.
To examine the convergence the reconstruction model, we compare the loss convergence curves of the D3G (Yang et al. 2023) model and our model. Figure [6 shows that the addition of the ReMatching loss does not affect the convergence behavior the optimization. We also show the loss curve of theRe.Matching loss itself in figure 17 is important note that for the ReMatching formulation, the optimal solution does not necessarily achieve 0 loss, simliarly to the APM procedure (Deutsch] [1922). Instead, it achieves the lowest loss possible given the reconstruction task and selected prior.
8.6 ABLATION FRAMEWORK HYPERPARAMETERS
23

---

Published as a conference paper at ICLR 2025
Ours
Baseline
Figure 14: Comparisons of the converged (white arrows) between the baseline and our approach that uses the ReMatching loss. The ground-truth velocities, ät or. are shown as green arrows. When the matched in aligns with the ground truth, the green arrows become indistinguishable.
0.055
0.05
Ours
0.045
D3G
0.04
time
0.035
0.03
0.025
0.02
0.015
10k
20k
30k
40k
number of Gaussians
Figure 15: Combined average time (seconds) for forward and backward pass for varying size of n.
In this section, we present an ablation study on key hyperparameters introduced by the ReMatching framework: i) the weight of the ReMatching loss, X, as defined in equation 8 ii) maximum number of parts selection, k for the adaptive prior class; and iii) the weight of the entropy loss (equation 29).
used to optimize the learned part assignments when employing adaptive prior class.
24

---

Published as a conference paper at ICLR 2025
0.5
loss (D3G)
loss (Ours)
0.4
0.3
0.2
0.1
0
10k
20k
30k
40k
iteration
Figure 16: Loss curves report of our model and D3G
Yang et al. 2023 over 40k training iterations.
0.12
ReMatching loss
0.1
0.08
0.06
0.04
0.02
10k
20k
30k
40k
itcration
Figure 17: Loss curve report for the ReMatching loss, showing running average with a window size of 20.
ReMatching loss weight. We note first that consistent value of 0.001 used across all scenes experimented with in section 6, already demonstrating the robustness this parameter. To further test this, conducted experiments the Hell Warrior and Lego scenes from the D-NeRF
25

---

Published as a conference paper at ICLR 2025
PSNR
PSNR
2003
2454
015
S9IM
SSIM
09421
3654
3.615
39915
LPIPS
LPIPS
0389
3832
0.91
2047
1049
Figure 18: Effect of the weight parameter on the PSNR evaluation metric for the Hell Warrior scene (left) and the Lego scene (right).
dataset, evaluating how different À values influence solution quality. Figure 18 shows these findings. We that for the Hell Warrior scene, employed the PIv prior class, while the Lego scene used the Py class. The results indicate stable improvement within the range of À 4, while small values s 5e-5 aligns with the baseline. Larger values, 1e-2, may compete with the reconstruction loss, leading suboptimal solutions.
Maximum number parts selection. To assess the impact the hyperparameter we selected the Mutant scene, which aligns with the adaptive prior class Pıv, and the Lego scene, corresponding to the adaptive prior class Py and evaluated how varying affects solution quality. Figure I9 presents the results of this analysis. The findings suggest relatively stable performance within the range k to = 15, offering flexibility in selecting based on leveraging prior knowledge about the expected number of moving parts in the scene. As a special case for k = the adaptive prior Py corresponds to Pr. If we train with ReMatching using this prior on the three selected scenes we get the following quantitative results: Lego (PSNR: 24.94 94, SSIM: 0.9536, LPIPS: 0.0452), Hell Warrior (PSNR: 40.77, SSIM: 0.9917, LPIPS: 0.0285), Mutant (PSNR: 41.88, SSIM: 0.9965, LPIPS: 0.0069).
Entropy loss weight. Similar to the À hyperparameter, the entropy loss weight was kept fixed across all experiments in section 6 To further examine its impact, evaluated its influence performance with varying weight values for the Hell Warrior scene. Figure 20 presents the results of this experiment, demonstrating stable performance within the range [1c-4, 1c-3].
8.7 ADDITIONAL QUALITATIVE EVALUATION
To further support the qualitative results presented in Figures and 3 the supplementary material includes additional evidence showcasing novel-view video reconstruction results. These compar- isons highlight the performance of our model relative to baseline approaches, providing a more comprehensive validation of its efficacy.
26

---

Published as a conference paper at ICLR 2025
PSNR
PSNR
11.7
3637
-1.5
2455
41.
3643
4.40
3632
SSIM
SSIM
1991
0395
3992)
89955
03010
LPPS
LPIPS
0.81
1005
00004
3644
Figure 19: Impact of varying & values on the PSNR evaluation metric for the Mutant scene (left) and the Lego scene (right).
PSNR
4145
41,35
SSIM
99923
88927
93930
89925
0954
1PIPS
1423
1:10
0.011
embodding weight
Figure 20: Impact of varying entropy loss weights the PSNR evaluation metric for the Hell Warrior scene.
27


Reviews:
[{"review_id": "rnTiATSCCv", "reviewer": "Reviewer_5yAB", "summary": "This paper tackles the challenge of dynamic scene reconstruction by introducing a novel loss function to incorporate diverse deformation priors into a unified framework. This new loss uses velocity-field-based priors and solves as a flow-matching problem. This design allows the framework to integrate a variety of deformation priors, such as piece-wise rigid transformations and volume-preserving deformations, enhancing the flexibility and generalization capabilities of existing models such as Gaussian Splatting. In experiments, the authors validate the robustness of their method across multiple benchmarks, demonstrating consistent improvements over baselines in both synthetic and real-world dynamic scenes.", "questions": "- Would it be possible to apply the new loss function directly to a baseline method without altering any other components? This would allow for a strong, fair comparison that isolates the effect of the proposed loss. For example, the authors can try conducting an ablation study where ReMatching loss is applied to a baseline method (e.g., D3G) without any other modifications, and report the results. This would help isolate the impact of the proposed loss function.\n- L215 \"we assume that P are linear sub-spaces, hence allowing for fast solvers with a runtime of at most O(n)\". Could the authors clarify why it is reasonable to assume P is linear and specify the fast solvers used? This additional detail would help readers understand the computational benefits and assumptions behind this approach.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces the ReMatching loss, designed to enhance dynamic scene reconstruction by leveraging deformation priors in a unified, adaptable way. This approach directly addresses limitations in generalization for dynamic NVS.\n-  ReMatching focuses on incorporating velocity-field-based priors to better model deformations over time, enabling seamless integration with existing dynamic reconstruction models. The framework supports various types of deformation priors, including restricted deformations, piece-wise rigid transformations, and volume-preserving deformations.\n- A notable feature of this framework is its capacity to combine multiple priors, allowing for the construction of more complex deformation classes. This adaptability is key for addressing the diverse requirements of real-world dynamic scenes.\n- By using the Gaussian Splatting image model as a backbone, the framework consistently outperforms existing methods across multiple benchmarks, demonstrating improved reconstruction accuracy and robustness in both synthetic and real-world dynamic scenes. It shows that ReMatching is not only theoretically sound but also empirically robust across various challenging scenarios.", "weaknesses": "- The proposed framework and loss function include some details and design choices, many of which seem crucial for achieving optimal performance. However, no ablation study is presented to isolate the effects of these parameters or design decisions, leaving it unclear how each contributes to the final results.\n- Introducing a new loss function raises questions about its impact on training stability and convergence speed, but these aspects are not explored in the paper. Additional insights into how this loss affects training dynamics would provide valuable context for readers, such as  runtime comparisons, convergence plots, or error analysis across iterations.\n- The paper asserts that the method is simulation-free, meaning that each evaluation of $\\psi_t$ requires only a single step, avoiding potential numerical instabilities. However, it's unclear if this claim has been experimentally validated or remains a theoretical assumption. If any simulation was conducted, sharing empirical evidence comparing the stability and computational efficiency would clarify the stability and feasibility of this approach.\n- From the quantitative results, the improvements over the second-best method are relatively modest. It remains unclear what the primary advantage of this approach is—perhaps robustness? Clarifying this would help readers understand the main strengths of the proposed method. \n- Additionally, there are numerous dynamic NeRF datasets and benchmarks beyond the synthetic, 360-degree dynamic NeRF dataset, such as those used in NSFF and DynIBaR, which feature more complex, real-world, forward-facing scenes. It would be insightful to see how well the proposed loss function performs on these more challenging datasets, as it would demonstrate its applicability to diverse, real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W2DKitND4k", "reviewer": "Reviewer_zqNk", "summary": "The paper introduces the ReMatching framework that provides deformation priors into dynamic reconstruction models. It can be adopted into multiple types or model priors and improve dynamic reconstruction accuracy over previous methods.\n\n---\n\nI rate the paper marginally below the acceptance threshold. The paper provides theoretical backgrounds and shows that it works, but it's not so clear what specific problem the paper is trying to tackle and overcome what challenges in the literature.", "questions": "* Hyperparameter choice\n\n   How is the method sensitive to the hyperparameter choice for the main objective loss and the entropy loss?\n\n* (minor)\n\n   Sometimes equations are referred to by just their numbers. In line 224, \"problem from 5 with 9\" $\\rightarrow$ \"problem from equation (5) to equation (9)\". It would be good to double-check and revise them between Page 3 and Page 7.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Theoretical background\n\n   The method provides theoretical background on its proposed representation. The supplemental further provides proof of lemma for better understanding.\n\n* Good accuracy\n\n  The method shows better accuracy than other previous works in general (depending on the evaluation datasets and metrics though).", "weaknesses": "* Killing application?\n\n   The method theoretically sounds good, but it seems not so clear what problem the method is mainly targeting. Are there any problems or failing cases that the proposed method solves whereas the previous method fails? Experiments (Table 1 and 2, Figure 2) show the benefit of the method for better rendering quality, but are there any major challenges that the method targets to solve?\n\n* Improvement in generalization quality?\n\n   The paper claims about the improvement in generalization quality, but the experiment shows that it doesn't clearly outperform previous methods, but is competitive or slightly better. In what sense does the paper improve generalization quality? I hope it's more clearly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B1rwCZCPry", "reviewer": "Reviewer_GNax", "summary": "This paper aims to solve dynamic reconstruction task. It proposes a new loss called ReMatching loss, in order to inject some physical priors into the learned deformation field. They also propose five categories of physics priors, and derive the corresponding forms of the RM loss respectively. The paper builds the model based on the deformable-3d-gaussian paper, and tests the performance for three of the five categories on both synthetic and real-world datasets, showing comparable performance to baselines.", "questions": "Apart from the questions listed in weaknesses, here are some more questions:\n\n1. Why the baseline numbers in the table not the same as the published version? According to the published version of D3G and GA3D papers, the performance of this paper is actually on the same level. For example, on D-NeRF datasets, D3G has 40.43 PSNR on average, while two types of the model have 40.36 and 40.54 on average. (lego scene is deleted due to data discrepancy, details in next question)\n\n2. The lego scene is known that data discrepancy exists in the scenarios presented in the training and test sets. This can be substantiated by examining the flip angles of the Lego shovels. So some works, like D3G, try to evaluate the model on the validation split because evaluating on the test split is meaningless. Which split is evaluated here? \n\n3. GA3D has reported its performance on HyperNeRF dataset, so why this baseline is deleted from the HyperNeRF experiments?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "In presentation, this paper shows dedicatedly derived loss form and clear derivation for different kinds of velocity priors. For the idea, the loss proposed by this paper constrains the function shape of deformation field with some physics priors. Working similar as a PINN loss, this framework seems flexible to different deformation designs.", "weaknesses": "1. In synthetic experiments, type IV and type V priors are selected for each scene. They should be thoroughly tested respectively to show the applicability under different scenarios. \n\n2. In real-world experiments, it is not clear which type of priors are used. And also, all of the priors should be tested to show the abilities. \n\n3. The velocity branches in the model only serve as a constraint part, but not used directly. The paper should demonstrate whether the learned velocity is meaningful or not. For example, using the velocity to advect the scene. \n\n4. The segmentation compared with SAM is not proper and confusing. Since SAM segments objects according to semantics but this paper segments according to motions, it’s not proper to punish SAM’s over-segmentation. Another question is about the bouncing ball example shown in the paper. As I know this scene includes some moving shadow on the ground, so the motion should not segment the shadow part as the same as the static ground. How is the segmentation applied?\n\n5. The performances are only comparable, it’s hard to tell whether the priors are really helpful or not. \n\n6. There is a lack of an illustrative figure to show the main architecture of the proposed method, so the reader can only know how the idea is working till the end of the paper.\n\n7. There are too many equations or derivations in the main text. Although they are clear and I appreciate the details, too many derivations in the main text make the paper lose its main focus. \n\n8. There are inconsistent baseline names in text and tables, for example, 3D Geometry-aware Deformable Gaussians is called DAG in main text but GA3D in the table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PAZ702gOcB", "reviewer": "Reviewer_SSgx", "summary": "This paper tackles the problem of dynamic view synthesis (with multi-view input). To model the time-dependent function, it introduces a flow-matching mechanism. Essentially, by assuming that a flow or velocity field exists that can reconstruct the true time-dependent functions with some predefined flow priors, e.g., rigid or volume-preserving, this paper utilizes the alternating projection method (AMP) to guide the training of the velocity field. Further, with some specific time-dependent function format, the authors demonstrate that there exist closed-form solutions to such AMP problems. Empirical experiments on D-NeRF and HyperNeRF datasets demonstrate the effectiveness of the proposed approach.", "questions": "1. Can authors clarify how to make HyperNeRF a multi-view dynamic view synthesis benchmark? Essentially, HyperNeRF only has two views. If the authors use up those two views, how to evaluate view synthesis results? Or do we only conduct time-interpolation evaluations? If the authors indeed carry out monocular dynamic view synthesis, please correct L100 as the \"multi-view images\" statement is not correct.\n\n2. In L131, the authors state that there is a reference function $\\psi_0$ to be pushed forward. Concretely, does $\\psi_0$ refer to the initialized Gaussian mean $\\mu^i$ in Eq. (29)?\n\n3. In L849, the authors state that they use **forward-mode** auto differentiation. Can authors clarify the reason of not using the traditional backpropagation, i.e., reverse-mode? Essentially, what prevents from using that?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- originality-wise: I appreciate the principled way that the authors formulate the problem of dynamic view synthesis as a prior-guided flow-matching problem. This will be useful and interesting to the community.\n- quality-wise: the approach presented in the paper is solid and the experiments demonstrate the effectiveness.\n- clarify-wise: overall, the paper is easy to follow.\n- significance: the task of dynamic view synthesis is of great importance to many downstream applications, e.g., AR/VR.", "weaknesses": "## Confusion about Eq. (9)\n\nI am quite confused with the definition used to derive Eq. (9). Concretely, the authors state $V = \\mathbb{R}^n$ in L202, which is an Euclidean space. However, the authors further define $\\psi_t  = (\\gamma_t^1, \\dots, \\gamma_t^n)^T$ where $\\gamma^i: \\mathbb{R}_+ \\mapsto \\mathbb{R}^d$ is a function. Do these conflict?\n\nFurther, I got confused about how Eq. (9) is derived based on the above definition. I tried to follow the product rule for the divergence, i.e., L212. However, I am unable to reach the result, especially $u_t (\\gamma_t^i)$. Can authors provide details?\n\n## Inadequate visualization\n\nSince the underlying mechanism is to train a velocity field $v$ (Eq. (2)) and its corresponding flow $\\phi_t$ (Eq. (3)), I am quite curious to know 1) how the reconstructed flow as well as velocity fields look like; 2) how the prior set $\\mathcal{P}$ (L153) look like; and 3) the difference between the prior and the trained flow. \n\nI think this is important such that readers can be well aware of the performance of the approach.\n\n## Speed\n\nCan the authors clarify the efficiency of the flow-matching approach? I think the main overhead compared to vanilla Gaussian Splats comes from the computation of the ReMatching loss $L_\\text{RM}$ (Eq. (7)).\n\nIf I understand correctly, in Algorithm 1, the $\\texttt{solve}$ step in L189 will be solved analytically based on those priors specified Sec. 4, which should be quick. Further, the computation of $\\rho$ in L190 should also be quick due to those priors. Can authors clarify whether this is the case?\n\n## Missed reference\n\nThe following work utilizes an ODE solver to conduct dynamic reconstruction from videos, which is quite relevant to this work.\n\n[a] Class-agnostic Reconstruction of Dynamic Objects from Videos. NeurIPS 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)