Neural Polynomial Gabor Fields for Macro Motion Analysis
Authors: Chen Geng
Hong-Xing Yu
Sida Peng
Xiaowei Zhou
Jiajun Wu
Venue: ICLR 2024 poster (2024)
Keywords: Scene Representation
Video Analysis
Motion Analysis
Neural Rendering
3D Vision
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a novel dynamic scene representation for macro motion analysis and editing.

Abstract: We study macro motion analysis, where macro motion refers to the collection of all visually observable motions in a dynamic scene. Traditional filtering-based methods on motion analysis typically focus only on local and tiny motions, yet fail to represent large motions or 3D scenes. Recent dynamic neural representations can faithfully represent motions using correspondences, but they cannot be directly used for motion analysis. In this work, we propose Phase-based neural polynomial Gabor fields (Phase-PGF), which learns to represent scene dynamics with low-dimensional time-varying phases. We theoretically show that Phase-PGF has several properties suitable for macro motion analysis. In our experiments, we collect diverse 2D and 3D dynamic scenes and show that Phase-PGF enables dynamic scene analysis and editing tasks including motion loop detection, motion factorization, motion smoothing, and motion magnification. Project page: https://chen-geng.com/phasepgf

Published as a conference paper at ICLR 2024
NEURAL POLYNOMIAL GABOR FIELDS FOR MACRO MOTION ANALYSIS
Chen Geng". Mong-Xing Yu' Sida Peng", Xiaowei Zhou?, , Wu' Sttanford University *Zhejiang University (gengchen, koven, jia jaajuuue@ec stanford. edu (pengsida, xwzhou e ju.jee....... edu.cn
ABSTRACT
We study macro motion analysis, where macro motion refers the collection of all visually observable motions in dynamic scene. Traditional filtering-basee methods on motion analysis typically focus only on local and tiny motions, yet fail to represent large motions 3D scenes. Recent dynamic neural representa- tions can faithfully represent motions using correspondences, but they cannot be directly used for motion analysis. In this work, we propose Phase-based neural polynomial Gabor fields (Phase-PGF), which learns to represent scene dynamics with Iow-dimensional time-varying phases. We theoretically show that Phase- PGF has several properties suitable for macro motion analysis. In our experi- ments, collect diverse 2D and 3D dynamic scenes and show that Phase-PGF enables dynamic scene analysis and editing tasks including motion loop detection, motion factorization, motion smoothing, and motion magnification. Project page: htttps://chee-ggenn com phasepe
INTRODUCTION
The physical world is dynamic and full of different motions: from waving trees to walking peo- ple, motions present in different spatial regions, have diverse magnitudes, and manifest various frequency characteristics. We refer to the collection of all visually observable motions a scene the macro motion. Factorizing and analyzing macro motions essential in understanding and interpreting the dynamic world. We argue that analyzing macro motions requires dynamic scene representation that bear three key properties: It should be able represent macro motions faithfully; it should enable decomposing macro motions in both spatial domain and frequency domain; and it should allow representing 3D scenes.
While modeling motions have been a constant topic of interest in computer vision, graphics, and machine learning, existing methods do not meet the three properties simultaneously. Traditional mo- tion processing and magnification methods allow analyzing motions by filtering imput videos in fre- quency domain and editing the frequency components' magnitudes (Wu et al. 2012) or phases (Wad- lwa et al. 2013). However, they only focus on micro motions (i.e., local and tiny motions) and they do not handle 3D scenes. Recent methods (Li et al. 2021 2022a Fridovich- Keil et al. (2023) exploit neural radiance fields (Mildenhall et al. 2021) to represent dynamic 3D scenes using deformation
fields (Pumarola et al. (022)) or flow fields (Du et al. 2021). For example, D-NeRF (Pumarola
et al. 2021) defines template neural radiance field in the canonical space and builds correspon- dences between the observation space and the canonical space using displacement field. Although 3D correspondences explicitly represent scene motions, they cannot be directly used for the motion analysis due to the lack of modeling underlying motion components.
In this work, we propose the basee neural polynomial Gabor fields (Phase-PGF) that simulta- neously meets all the three key properties for macro motion analysis. In particular, Phase-PGF rep- resents a dynamic scene as a composition of wavelet-based neural fields, and the wavelet basis modulated by set of temporally- canporallyvvrrying phases. Therefore, macro motions in dynamic scenes are captured by the phases. We show that under appropriate assumptions, Phase-PGF has theoret- ical properties that allow various macro motion analysis tasks, such as motion separation, motion
"qqual contribution.

---

Published as a conference paper at ICLR 2024
smoothing, and motion intensity adjustment. We then instantiate Phase-PGF with novel neural architecture and a training scheme for higher-quality dynamic scene representation and editing.
In experiments, collect examples of both 2D videos and 3D dynamic scenes (represented by multi-view videos). We show that Phase-PGF allows macro motion analysis on both 2D and 3D dynamic scenes, allowing several macro motion analysis applications including motion loop detection, motion separation, motion smoothing, and motion magnification.
In summary, our contributions threefold: Firstly, we propose and formulate the problem of macro motion analysis. Secondly, we introduce Phase-based neural polynomial Gabor fields (Phase-PGF) for macro motion analysis. Lastly, our experiments show that Phase-PCi PGGse allows series of dynamic scene editing tasks on both 2D and 3D scenes.
RELATED WORK
Dynamic neural representations. Using neural representations (Mildenhall et al. 2021 Sitz- mann et al., 2020) to represent dynamic signals has emerged as popular research topic in re- cent years. It has been widely used in representing 4D (3D) space and 1D time) dynamic volumet- ric videos with advantages including high-fidelity rendering and low storage requirements. Some methods build explicit correspondences between frames by extracting dense flow or deformation (Pumarola et al., [021] Du et al. 2021 Li et al. 2021 2023b Park al. 2021a b). However, such representation for motion is dense and is hard to analyze. Other approaches use inductive bias on al. 2021 2023 Weng et al.) 2022 yet cannot be applied to general dynamic scenes. Recently some approaches propose to use hybrid representations (Fridovich-Ked et al., 2023 Shao et al., 2023) to implicitly represent dynamic 31) scenes, lacking interpretability on motion information. Beyond 3D dynamic scenes, there are also works focusing on using neural representations for 2D videos al. 2022b Chen et al. [02T Zhang et al. 2021b] Rho et al.) 2022). However, most of them use implicit motion representation which cannot be easily analyzed or edited.
scenes to assist the reconstruction of dynamic scenes(Peng
Motion analysis and editing, Motion analysis and editing in dynamic content is an important task computer vision. Phase information is widely used in the field of motion analysis (Gautama & Van Hulle 2002; Meyer et al. 2018;) Mai & Liu 2022). For motion editing, early work solves this problem with a Lagrangian perspective(Lius et al. 2005), yet renders artifacts with large motions. Wu al. (2012) proposes to understand this task with Eulerian perspective and Wadhwa et al. (2013) further proposes a phase- based method to perform motion magnification. Some other works (Davis et al. 20155 a Davis 2016 use physical modal analysis to manipulate local motions. Despite being successful for tiny motions, they cannot make good editing on large and macro motions studied in this paper.
Interpretable and editable neural scene representations. With the development of neural rep- resentations (Mildenhall et al. 2021 Sitzmann et al., 2020), the interpretability of such methods has become an important research area. Several works on this topic focus on the frequency domain or the coarse-to-fine decomposition of the input signal (Lindell al. 2022 Yang al. 2022 Martel al.,
2021) Other works target at spatial decomposition of the input content (Shuai et al., 2022 Zhang al. 2021a| Lu et al.| 20200). allowing modifying the scene contents at an instance level. These prior works primarily solve static scenes, yet our work focuses on dynamic content. Moreover, our work focuses on the interpretability from the macro motion perspective of the representation, differing from prior works.
Concurrent works. Recently there have been some concurrent works on tiny motion analysis and editing. |Feng et al. (2023) proposes method to magnify local and tiny 3D motion, yet their method has challenges in manipulating macro motion discussed in this work. [i al. (2023a) gives an approach to manipulate local motion, but their focus is also tiny motions.
3 APPROACH
In this section, we first formulate the problem of macro motion analysis. Then, we show an ab- stract formulation and theoretical properties of the proposed Phase-hased neural polynomial Gabor
2

---

Published as a conference paper at ICLR 2024
fields (Phase-PGF). Next, introduce how instantiate the formulation with neural networks to represent 2-D and 3-D dynamic signals. Finally, we present a training scheme for our approach.
3.1 MACRO MOTION ANALYSIS
We define the macro motion as follows:
Definition 3.1 (Macro Motion). We assume a dynamic scene can be decomposed into k components
{st, 82.-. ;Sk; ) each with rigid motion (t).). The dynamic scene is represented using the following implicil function:
(x s(x,1) Σ (
+x(xx
(1)
where we assume (x) is differentiable and non-constant over its domain. The macro motion is defined as y y2 (t))).
Note that this definition does not imply that cognitive object is only represented using only one component 8,. Rather, an object can be represented using several components with several rigid motions, making it possible to model complex scenes without loss of generality.
are interested in analyzing macro motion. Specifically, we aim to find a low-dimensional repre- sentation that represents y and can be factorized in both the spatial domain and frequency domain.
3.2 PHISEE-PGF: PIAASE-BSSED NEURAL POLYNOMIAL GABOR FIELDS
We propose Phase-based neural polynomial Gabor fields (Phase-PGF), In particular, to inherently allow factorization of the represented signals in both spatial and frequency domains, we adopt poly- nomial neural fields (Yngg et al. 2022) with Gabor basis functions (Feichtinger & Strohmer 2012) as the basic building block. To represent motions a fow-dimensional space, we leverage the phases of the Gabor basis functions. The formal definition is given by:
(91, 92, ... 9m to shift-orthogona set of Gabor func- tions (x; The P4, We, (b) = exp( 3/1 ||x (bl12) sin(wix + (h). Let H = (h1 ((()) .... hm (t)) be a set of phase functions where (t,()) IR_ + R.. A Phase-based neural polynomial Gabor fields Phase-PGFF) is a neural network f fxx t) = PL PPL-1 0 P1 (xx t). where Vi, ps are finite de- gree multivariate polynomials, and (x = (91 (x h (t)), 92 (t)), 99 (t)3
Definition 3.2 (Phase-PGF).
is m-ddmmensional feature encoding using basis B and phuse functions H.
Intuitively, the time-varying phase information allows our model represent the maero motions using a dow-dimensional vector. Therefore, the macro motions can be analyzed and manipulated by controlling the phase information.
We then theoretically discover the properties of the proposed representation. First, we show that the phases can be used to analyze the periodic components the macro motion in the scene.
Theorem 3.1 (Periodicity Correlation). For Phase-PGF with phase functions HH already trained to represent a dynamic scene s(x, t), if there exists T such that some motion (t yt))), then 3hg E H.,, T) = (t).).
We include the proofs of all theorems in appendix A This theorem indicates that periodic motion components can correspond components of the phase information.
If scene has complicated composed motions, e.g., a scene with multiple dynamic objects, Phase- PGF allows unsupervisedly factorizing different motion components into different components in the phases. Specifically, we have the following property:
Theorem 3.2 (Motion Separation). Assume that a scene has several objects with motion y:(1) sociated with each object i. For each component, the implicit functions representing object lies in
(x + a). g, B)) 5yg where 8 is the Kronecker delta function.
3

---

Published as a conference paper at ICLR 2024
WM WV
Phase Generator
Feature Map
mm
Decoder
Phases
Gabor Basis
Polynomial Gabor Fields
Modulate
Rendering
Figure 1: Overview of Phase-PGF. A phase generator yields üime-varying phases to modulate the Gabor basis in Phase-PGF, which produces a latent feature field for volume rendering. Please refer to the text in Sec. for more descriptions.
the span of B. Then, a Phase-PGF f(x, t) can be decomposed into several Phase-PGFss
f(x,t) = -E((xxt)
where (x, a Phase-PGF with basis B..
This theorem indicates that is possible disentangle the scene macro motion based on the phases. For example, if the macro motion can be separated into low-frequency components,
components and the macro motion contains motion components present different spatial regions, then by filtering out the desired phase components, we can manipulate the scene motion factorized manner.
We have now shown that the extracted phase information from the Phase-PGF can be used to observe the properties of the macro motion. We then show that this representation can be edited to manipulate the scene motion.
Theorem 3.3 (Motion Smoothing). Assume that the a dynamic scene is represented by a trained Phase-PGrs PGF with phases H. If we apply low-pass filter on some hy that corresponds to some motion ys, then the motion would be smoother, i.e., y:)( is attenuated.
Intuitively, motion smoothing corresponds to attenuating sighfreeuencc motion components. Be- sides motion filtering, is also possible to perform the manipulation in the dimension of motion intensity/magnitude:
Theorem 3.4 (Motion Intensity Adjustment), Assume that scene component motion takes the form (t) = ex and the scene is represented using Phase-PGF with phase hy(t) corresponds to (t). ) If we multiply the phase by a coefficient A, then the scene motion would be manipulated to yi(t) Ae
We include the proofs of all the theorems in appendix
A
3.3 INSTANTIATION PHASE-PGF
In the previous section, we have introduced the abstract definition of Phase-PGF and its properties. In this section, we further illustrate how we empirically instantiate Phase-PGF to represent 2D and 3D dynamic scenes. We show overview in Figure | where we have phase generator (on the left of Figure 5) that generates the m time-varying phases H. = [h1 (t), hz((), ... hm)(t) (t)) for modulating the Gabor basis B {91. 92. ... . .m }. The Phase-PGF f(x, PL PL-1 o....pp V (xx, t) generates latent feature map and a neural latent decoder (detailed below) renders an image from the feature map.

---

Published as a conference paper at ICLR 2024
Phase Generator for H. Since the phases central to our model, the phase generator architecture should ideally be expressive and structured. To this end, we instantiate each h2 with a Polynomial Neural Field (PNF) (Yang al. 2022). which allows decomposing the phases into frequency sub- bands. The frequency decomposition of phases allows easy manipulation over the phases, thus over the macro motions.
(x, Neural Latent Decoder. While Phase-PGF takes the form of f(x, t) = Pr. P o...........................oo op o V (x, t) where p denotes a polynomial, we empirically find that consumes a large amount of GPU memory if we directly use fxx !) to represent raw pixels in complex and high-resolution scenes. Therefore, we propose to couple Phase-PGF with a neural latent decoder. That is, Phase- PGGs represents a low- resolution latent feature field of the dynamic scenes, and the neural latent decoder decodes the latent features into high-resolution results, similar to the feature fields in prior works (Niemeyer & Geiger 2021
More specifically, for representing 2D videos, we first use Phase-PGF f(x, t) to render 2D feature map m. Then, we use a 2D CNN decoder to render the final image I = D (m). For 3D dynamic scenes, to render frame from viewpoint, formation. That is, we do ray marching to get a set of sampled points on the rays of the frame, and then we use volume rendering to aggregate the sampled features to form the feature map. We also use a 2D CNN to decode the feature map into a final RGB image.
adopt the volume rendering (Mildenhall et al.) 202I
The neural latent decoder D is designed in a pyramid manner. We first decode the feature into low-resolution RGB image J. = (r(m). Then, we use another decoder Du to obtain the final high-resolution output f = DH (TL).
We refer the readers to appendix C.I and appendix C.2 for more details on the neural network implementation and the phase space.
3.4 TRAINING PHASE-PGF TO REPRESENT DYNAMIC SCENES
Multi-stage Training, We train Phase-PGF in a multi-stage manner. In the first stage, we supervise both the learning of Phase-PGF F and decoder Dr with reconstruction loss L1 = 1| where 7. is the ground-truth low-resolution image.
In l12.
In the second stage, we further improve the high-resolution detail of the rendered image with patch- based sampling and perceptual loss (Johnson et al.) 2016), during each training step, randomly
sample patch from the high-resofution image, and the loss La is defined as La || /112 + [magg ()) magg (2),,, where is the patch from the ground-truth high-resolution image and mygg is a feature map extracted by a pre-trained VGG network (Johnson et al. 2016).
Adversarial Training. The multi stage training enables high-fidelity reconstruction of the input scenes. However, extrapolating macro motions (e.g., for motion magnification) requires further regularization on the neural latent decoder. Therefore, we propose to further use adversarial training after the stages. In particular, given the generated phases H. we perform simple magnification to it and apply these augmented phases to the decoders. We want the decoder to generate plausible images even when the input phases are extrapolated. Specifically, for a given phase sequence h(t) H. the augmented h (t(t),, bi, bu) is defined as follows:
h' (t(t),, bi, bn) = h((t) + (A - 1) v(tlbr, bn). y(t[br, bn)) = FF
(3)
bi, bw)),
(4)
where X is intensity manipulation coefficient, by and by are subband limits for specific component of the signal, F and F-1 are fourier transform and its inversion, and T(f) is band-limit filter defined as follows:
T(f) = f1, ifisss S2.
(5)
otherwise.
We additionally construct a discriminator D to judge whether the image decoded from the extrapo- lated phase sequence is similar to the input sequence. The adversarial loss is defined as follows:
Lade [oog D( 1)] + [xog( D(Tr,x(x, 1)),
5

---

Published as a conference paper at ICLR 2024
Ma
6.-
-02
-0.4
NMM
10
30
50
60
70
Phase-NIVR
0.30
9.20
-0.55
35
(b) Zoom-In View and Visualization of Corresponding Frames
(a) Comparison
Figure 2: On the interpretability of motion representations of Phase-PGF and Phase-NIVR. (a) Given the 2D videos in (b), we compare the representative phases generated by our method and Phase-NIVR. (b) We zoom in on the phase generated by our model and visualize some keyframes illustrate the correspondence between the macro motion and the generated phase.
Sequence
Clock
76.5%
Ours
Phase-NIVR (Mai & L.u 2022
Pendulum
81.4%
Balls 90.2%
Dumping 0.999
0.015
Projectile (X) 0.998
0.793
Bouncing 76.5%
Projectile 0.859
0.066
Average
79.2%
Table 1: Human preference study the interpretability of generated phases. The phases from ours are more interpretable than the baseline.
Quantitative comparison using normal- ized cross-correlation datasets. Projectile(X) and Projectile (Y) mean motion in two different spatial dimensions.
Table 2:
phases in synthetic
where T denotes a functional that augment the phase generated by with a motion intensity adjustment coefficient X, as defined above. For more details on training, please refer to appendix C.4
EXPERIMENTS
To evaluate our approach on the task of macro motion analysis, we collect several examples in- cluding both 2D and 3D dynamic scenes. We show that our Phase-PGF learns interpretable motion representations to allow macro motion analysis, which is demonstrated by motion loop detection and motion separation. Afterward we show that Phase-PGF allows macro motion editing tasks including motion intensity adjustment and motion smoothing.
4.1 EXAMPLES
We collect several dynamic scene examples including both 2D and 3D scenes.
Synthetic 2D videos. We render several different videos with varying motion complexities with ground truth motion quantitatively evaluate the proposed method. Please refer appendix [.]

---

Published as a conference paper at ICLR 2024
Frequency
Y-T Plot Original Motion
Y-T Plot of the Loop Video
FFT of Generated Phuse
(1) Input
(d) Phuse Gemeruted
Figure 3: Loop detection using extracted phase information. Given the (a) input video, we first learn the (d) phase represent the macro motion. We then perform (e) FFT on the phase get the frequency domain on the phase and get the periodicity information. Then the phase was segmented
based on the period which is shown as different colors in (d). The seamless video (c) can be then made according to this information and we show the corresponding Y-T plots in (b) and (c).
y
(a) Dynamic Scene
(c) Original Y-T Plot for the Red Ball
(c) Original Y-T Plot for the Blue Ball
y
y
(b) Phase Generated
Manipulated Y-T Plot the Red Ball
( Manipulated Plot
for Blue
Figure 4: Motion separation. This 3D scene contains two balls with different motions. Our method is able to decompose their motion using different phases, shown in (b). We attempt to show that we are able keep the red ball's motion unchanged while modifying the motion of the blue ball. (c) and (d) show that the motion of the red ball remains unchanged after modifying. (e) and (f) show that we successfully attenuate motion of the blue ball.
Real 2D videos. We also collect several real videos from the Internet and from our own capture to show the generalizability and applicability of the proposed Phase-PGF. We take some videos from Mai & Liu (2022) and other Internet sources to form dataset of real 2D vidcos.
3D Dynamic Scene. Our Phase-PGF can also be used model 3D dynamic scenes with neural rendering. To demonstrate this, we render synthetic 3D dynamic scenes with two different balls bouncing on a table top.
4.2 MACRO MOTION ANALYSIS: LooP DETECTION AND MOTION SEPARATION
For macro motion analysis, a preliminary requirement is that the motion representation is inter- pretable, i.e., the components of the factorized motion representation should be able to establish correspondences to scene motion components. Phase-PGF against previous method Phasc-NIVR (Mai & Liu, 2022). Phase-NIVR also aims to generate phases for input motion, yet it does not allow representing 3D scenes. We show a compar- ison of phases by both methods in Figure (a), and show keyframes corresponding our generated phases in Figure (b).
do this, we compare the phase generated by our
From Figure 2 observe that Phase-PGF generates phase sequence that aligns with the motion more coherently. A possible reason is that Phase-NIVR uses the SIREN network (Sitzmann al. 2020) as the representation which does not support decomposing different components

---

Published as a conference paper at ICLR 2024
Original
Ours
Phas-NIVR
Input
Wacchea of 2013
P-o
Keyframe
Keyframe
Keyframe 3
Figure 5: Comparison of the result of motion intensity adjustment between different baselines and the proposed method. In the first column, we show the different X-T plots generated by concatenating the strips indicated by the blue line in the input video. In the following three columns, we visualize the rendered results different keyframes of the input motion.
of the dynamic scene. Please check out the supplementary website at /tttttt:// // //tp:/::/cheenggng com/phasepgE for video results.
To quantitatively evaluate the motion representuation interpretability, we conduct a human prefer- ence study. We show several videos and the generated phases by both methods, and ask the partic- ipants to pick one that they believe better explains the motions in the videos. We use a research- oriented cloud source platform") We show the human study results in Table From Table we that the phases generated by Phase-PGF are believed to better explain the motions.
We additionally evaluate the normalized cross-correlation of phases generated by method and the baseline on datasets with ground motion trajectory. The results are shown in Table Please refer appendix D.2 for more details this.
To discover more complex motion patterns, we synthesize and collect several videos with varying motion complexities and evaluate them in appendix D.1
We further show two macro motion analysis applications: motion loop detection and motion sepa- ration.
Motion loop detection. In Figure 3] we show an example of motion loop detection on the input video. To do this, identify the periodicity of the input video, and then perform phase fus- ing Maa & Liu 2022) to generate an infinitely looping video.
synthetic 3D scene, where two balls bounce at different speeds. Phase-PGF is able to decompose the macro motion using a fre- use band-limit filters on the generated phase sequence, and then
Motion separation. In Figure 4 show motion separation
quency filter. Specifically, isolate the motion of interest (the motion of the blue ball) according to the frequency. In this way, we can manipulate (mollify) the motion of the blue ball without affecting the motion of the red ball.
4.3 MACRO MOTION EDITING: INTENSITY ADIUSTMENT AND SMOOTHING
Besides analysis, our Phase-PGF also allows editing macro motions. Specifically, we consider two motion editing tasks: motion intensity adjustment and motion smoothing. Please refer to ap- pendix for details on the editing operations.
Thii human preference study form be found the appendix.
8

---

Published as a conference paper at ICLR 2024
(a) Input Video
(b) Original X-1 Plot
(c) Smoothed X-1 Plat
Figure 6: Motion smoothing. We smooth the motion in the video by removing high-frequency components in the phase. (b) and (c) shows the X-T plot of a slice in the original and edited video.
Ours
Phase-NIVR
Wadhwa et (0133)
Motion Magnification Score
3.30
3.06
2.24
Visual Quality Score FID
3.14
1.76
1.88
0.512
10.255
5.889
Table 3: Quantitative comparison on the result of motion intensive adjustment. For the user study part (First two rows) We ask the user to give two scores for each video generated by the corresponding method. The first score is asked evaluate how large the visual motion magnification intensity is (1 for identical and for the largest). The second score is asked to evaluate the visual quality of the rendering (I for the worst and for perfect). We additionally evaluate the FID score for the videos generated by different methods, as shown in the third row.
Motion Intensity Adjustment. Phase-PGF allows modifying the intensity or magnitude of the macro motion in the represented dynamic scenes. Following the method introduced in Sec. 3.4 we show motion magnification results in Figure 5 We compare our method with traditional mo- tion magnification method Wadhwa et al. (2013) and Phase-NIVR. From Figure 5 we observe that Phase-PCC PGF allows magnification with much fewer artifacts. In comparison,
NIVR generates
a temporally flickering video with ghosting artifacts, and Waddwa al. (2013) failed magnify the macro motion as it focuses on tiny motions. Our observation is further consolidated by a human preference study, where we show the results in Table
We also evaluate the Fréchet Inception Score for manipulated videos generated from different meth- ods. The result shown Table It can be seen that our method surpasses other baselines greatly in terms of visual quality. Please refer to appendix D.2 more details.
We perform ablation studies to prove the effectiveness of the introduced components. Please refer to appendix D33 for detailed discussion.
Motion Smoothing. We show that Phase-PGF also allows motion smoothing. Figure we show that by removing high-frequency bands of the generated phase sequence, we can perform motion smoothing on the macro motion. From Figure 6. we observe that the high-frequency flickering of the paddle has been removed, lcading to smoother motion.
CONCLUSION
In this work, we propose and formulate the problem of macro motion analysis. We propose Phase- based neural polynomial Gabor fields (Phase-PGF) that represents motions in dynamic scenes with generated phase sequences. We show that Phase-PGF allows multiple macro motion analysis and editing tasks, including loop detection, motion separation, motion magnification, and motion smoothing.
Limitations. One limitation is that Phase-PGF shows slight artifacts in boldly magnifying large motions. Another limitation that Phase-PGF currently does not scale well to complex large-scale 3D dynamic scenes due to computational efficiency (we need more Gabor basis in larger scenes). This might be addressed by spatially adaptive Gabor basis. We also discuss other failure cases at appendix D.7
ACKNOWLEDGMENTS
This work is in part supported by the Stanford Institute for AI (HAI) and Sam- sung.

---

Published as a conference paper at ICLR 2024
REFERENCES
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neu- ral representations for videos. Advances in Neural Information Processing Systems, 34:21557- 21568, 2021.
Abe Davis, Katherine L Bouman, Justin Chen, Michael Rubinstein, Fredo Durand, and William Freeman. Visual vibrometry: Estimating material properties from small motion in vidco. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 5335-5343, 2015a.
Abe Davis, Justin G Chen, and Frédo Durand. Image-space modal bases for plausible manipulation of objects in video. ACM Transactions on Graphics (TOG), 34(6):1-7, 2015b..
Myers Abraham Davis. Visual vibration analysis. PhD thesis, Massachusetts Institute of Technol- ogy, 2016.
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14304-14314, IEEE Computer Society, 2021.
Hans G Feichtinger and Thomas Strohmer. Gabor analysis and algorithms: Theory and applica- tions. Springer Science & Business Media, 2012.
Brandon Feng, Hadi AlZayer, Michael Rubinstein, William T. Freeman, and Jia-Bin Huang. Visu- alizing subtle motions with time-varying neural fields. In International Conference on Computer Vision (ICCV), 2023.
Sara Fridovich-Ked Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12479-12488, 2023.
Temujin Gautama and MA Van Hulle. A phase-based approach to the estimation the optical flow field using spatial filtering. IEEE transactions neural networks, 13(5):1127-1136, 2002.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part 11 14, pp. 694-711. Springer, 2016.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5521 2022a.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6498-6508, 2021.
Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics. arXiv preprint arXiv: 2309.07906, 2023a.
Zhengqi Li, Qianqian Wang. Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neu- ral dynamic image- based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4273 4284, 2023b.
10

---

Published as a conference paper at ICLR 2024
Zizhang Li, Mengmeng Wang, Huaijin Pi, Kechun Xu, Jianbiao Mei, and Yong Liu. E-nerv: Expe- dite neural video representation with disentangled spatial-temporal context. In European Confer- on Computer Vision, pp. 267-284. Springer, 2022b.
David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVE con- ference computer vision and pattern recognition, pp. 16252-16262, 2022.
Ce Liu, Antonio Torralba, William T Freeman, Frédo Durand, and Edward H Adelson. Motion magnification. ACM transactions on graphics (TOG), 24(3): 2005.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33: 1525-11528,, 2020.
Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin, William Freeman, and Michael Rubinstein. Layered neural rendering for retiming people in video. arXiv preprint ar arXiv:2009.00 07833, 2020.
Long Mai and Feng Liu. Motion-adjustable neural implicit video representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10738-10747, 2022.
Julien Martel, David Lindell, Connor Z Lin, Eric Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:2105.027888 2021.
Simone Meyer, Abdelaziz Djelouah, Brian McWilliams, Alexander Sorkine-Hornung, Markus Gross, and Christopher Schroers. Phasenet for video frame interpolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 498-507, 2018.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453-11464 2021.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 5865-5874, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven Seitz. Hypemerr: A higher-dimennsional representation for topologically varying neural radiance fields. arXiv preprint arXiv:21066.10 13228. 2021b.
Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14314-14323 2021.
Sida Peng, Chen Geng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Implicit neural representations with structured latent codes for human body modeling, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10318-10327, 2021.
Daniel Rho, Junwoo Cho, Jong Hwan Ko, and Eunbyung Park. Neural residual flow fields for efficient video representations. In Proceedings of the Asian Conference on Computer Vision, pp- 3447-3463, 2022.
11

---

Published as a conference paper at ICLR 2024
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assiser Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241. Springer, 2015.
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Ten- sor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16632-16642 2023.
Qing Shuai, Chen Geng, Qi Fang, Sida Peng, Wenhao Shen, Xiaowei Zhou, and Hujun Bao. Novel view synthesis of human interactions from sparse multi-view videos. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473. 2020.
Neal Wadhwa, Michael Rubinstein, Frédo Durand, and William T Freeman. Phase-based video motion processing. ACM Transactions on Graphics (ToG), 32(():1-10, 2013.
Chung-Yi Weng. Brian Curless, Pratul P Srinivasan, Jonathan Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition, pp-
16210-16220, 2022.
Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William Freeman. Eulerian video magnification for revealing subtle changes in the world. ACM transactions on graphies (TOG), 31(4):1-8. 2012.
Guandao Yang. Sagie Benaim, Varun Jampani, Kyle Genova, Jonathan Barron, Thomas Funkhouser, Bharath Hariharan, and Serge Belongie. Polynomial neural fields for subband decomposition and manipulation. Advances in Neural Information Processing Systems, 35:4401 14115, 2022.
Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable fire-vvewppint video using layered neural representation. ACM Transactions on Graphics (TOG), 40(4):1-18, 2021a.
Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression, arXiv preprint 12. 1312, 2021b.
Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In ICCV, 2023.
12

---

Published as a conference paper at ICLR 2024
A DEFINITIONS AND PROOFS
In this section, we provide proofs of the theorems in the main paper. Our Phase-PGF is based on the polynomial neural fields (Yang et al. 2022). In the following proofs. we will use some of the properties of the polynomial neural fields. We refer the reader to Yang et al. (0022) for more context.
Proof of theorem 3.7 Let's assume that the scene at 0 is represented with the implicit function s(x), and the active content is doing rigid motion yt)). Then the dynamic scene to be represented can be denoted as s(x + y()), where both s(x) and y() are unknown.
Now the dynamic content is fit using Phase-PGF fxx, L), which gives:
f(x,t) s(x + y(t), Vx,t.
we have y(t + 7) = (1) holds for any L, then:
1(x,+ = o(x+y(1) TT)
(8)
=s(x+y(1))
(9)
= [.r
(10)
Consider the property of Polynomial functions Yang et al. 2022), we have:
f(x, t) = Σ wi (x + h,(t)),
(11)
then we can have:
Σ + (x + h,(t)) = 0,
(t(++
Vx, t.
(12)
Consider doing inner-prodee with 9j. we have:
Σ wi (s h+(t+T)) g:(x - (x(t))),
(13)
m
Σ wxlgi( - h,(t + 9T),, - (g.(x + 9x)))),,(x)))
(14)
wylg (x + (t + T)) g (1))).,;;x)) (x))
(15)
0
(16)
Therefore, we have (> + ht + T)) = 9₁ (x + ht)))), which implies h,(t + T) = (t)
n
According to the assumption, the dynamic scene can be represented using the following implicit function:
Proof of theorem 3.2
g(x, 1) = - (x
(17)
where 8; is in the span of B, Also, have:
13

---

Published as a conference paper at ICLR 2024
Σ
sx y.(() - f(x,t).
(18)
Considering the property of polynomial fields Yang et al. 2022 the f(x, t) can be written as:
m
f(x,t) Σ
wigi( + +h,(t),,
(19)
Then we have:
m
(5 + v:(1)) - Σ wigi(x + h:(1))
(20)
wigh(x + (.(1))
(21)
Thus we can get:
+x = Σ wigs(x ht))),, Vi EB
(22)
Let's denote fx t)
Eg,EB, Wigr(x, (x, t), then we have got a decomposition of f(x, t)
Ei(x,, t), where each f; is supported by basis B,
Proof of theorem 3.2 According to the assumptions, if the rigid motion is represented using (t), then:
s(x + y(t) f(x,t),
(23)
Σ wigi + (t)),, Vx, f.
(24)
We have used the property of the polynomial fields et al., 2022) in the derivation above. We first perform Fourier decomposition implicit function sxx) and the basis (x)), so that we can get:
s(x) / G(ffettfed
(25)
(x(x) [ g((f)e
(26)
By modulating the equation above, we can obtain:
s(x +(x+ G(f)e 2n
(27)
Σ wigi(x hh(()))s
(28)
Σ Wi Gi(F)e 22 Vx.t
(29)
We perform the Fourier transform of the equation above.
14

---

Published as a conference paper at ICLR 2024
G(f)e (2fu(() = Σ (i Af.
(30)
Then we have:
v(()
2ri5 In G(J)
Σ (e
(31)
By performing low-pass filter on ht(t), we can obtain new phase (t) with attenuated high- frequency information, which has the property:
hr (t) - di
(32)
We consider the derivative of the corresponding yı to evaluate the change in its smoothness.
dt
d
vr) dt 2
=
d
In
G(f)
Σ with
(e
(33)
d
In|
1
Σ Weei(f)ee22shall
(34)
2mi dt
G(f)
G(f)
dt G(f) Σ (f (225h220)))
d 1e
2m2f. Σ [f
(35)
d
2 E (f dt (f
(36)
= 2mif. E (i
Σ (f) e c224/bal0||
(37)
dt
= 2mif.. E Eung(F))eaatho
wigg (2) (2m (2)
(38)
dt
Σ (f) d di ht)l
(39)
Eurg. (f)
Σ (f) (t,(t))
<
(40)
I Ewi
1 v01)-
(41)
Therefore, we can have that the yu (t) is smoother than yt)), which finishes the proof.
Proof of theorem 3.4
According to the assumption, we have:
s(x + ctt)= = f(x,t) t)
(42)
(x (t)))
(43)
Since assume sxx) is in the span of B, it can decomposed into:
s(x) = Σ (x)
(44)
Then:
15

---

Published as a conference paper at ICLR 2024
Engi + + Σ wigi(x + (oigi(++1100 ht(t))
(45)
According to the shift-orthogonaliyy of B
h2(() = e
(46)
If perform motion intensity adjustment to (c) with magnitude A, then get modified phase h' (t) = Ah;(t) so that the scene motion can be modified to:
11(() - Ah;(()
(47)
Ae
(48)
B HUMAN PREFERENCE STUDY
We conduct user studies using the Prolific platform.
For the user study shown in Table D we have collected 102 effective responses. We designed our form using Google Forms. In the form, we asked the user to compare pairs of videos with plots try- ing to explain the motion inside the video. For each pair, we asked the user the following prompt: In which video (left or right), does the bottom plot better explain the motion in the top video? The user will choose between two answers and we collect the data and make statistical analysis. The video pairs can be found at our supplementary website: https: wwtps///ceennggnn...e www.enggeggcommmphseeeeee
For the user study shown in Table 3 we have collected 110 effective responses. In the study, we first asked the users to watch three video pairs from three methods (each pair consists of a top video and bottom video). In each pair, the bottom video is trying magnify the motion in the top video. Then for each pair, we asked them two questions. The first question is: The bottom video tries to magnify the motion of the white object in the top video. Do you think the bottom video successfully magnifies the motion? Please rate on scale of to 5, where means "Not at all" and means "Very much so". The answer to this question
used to calculate the Motion Magnification score. The second question is: You are viewing the same video pair as before. Please evaluate the visual quality of the bottom video. On scale of I 5, rate the quality of the bottom video. The answer to this question was used to calculate the Visual Quality Score. The videos can be found at our supplementary website:
www.co///pc..... phaapph
C METHOD DETAILS
In this section, we provide more details on the proposed method. The code will be released upon publication.
C.I NETWORK ARCHITECTURE
We first elaborate on more details on the network architecture. In the following, we discuss specific design choices related to different modules in the proposed framework.
Phase Generator. The key aspects an ideal phase generator are expressiveness and controllabil- ity. To achieve these two goals, t to serve as the backbone. Specifically, we follow the architecture described in Yang et al. 2022) where the basis is defined using Fourier waves.
a polynomial neural field (Yang et al. 2022) conditioned on
Each polynomial neural field takes normalized timestamp as input and outputs one- miedimension scalar h() the phase in the timestamp f. Following Yang et al. (2022). h(()) is defined as below:
16

---

Published as a conference paper at ICLR 2024
Hyper-Parameter Name
Value
Output Dimension
I
Hidden Dimension
8
Number of Bandwidths 4
Table 4: Syper-parametery used in Phase Generator.
h(t) Ff0)
(49)
where (t) is sub-PNF for one subband. (t) further implemented using the following factor- ization:
Et(t) tanh Σ
(50)
k=1 Zz,1(t) = An)) Zz,(t)) = G.
(51)
(52)
where G1(,, a, b) is subband limited in R() (a, b, d(O)), 8). As b. - b Gz(4,,b) is expressed in the linear combination of basis sampled from the defined subband:
b) Wang(() e (a, 8) W. R
(53)
where "/ is Fourier basis.
Each polynomial neural field h(t) represents continuous one-dimensional phase. To represent the whole phase space of dimension K, we instantiate an individual polynomial neural field ht for each phase in the phase space. Then ensemble them to form the whole phase.
We list the hyper-parameters used to instantiate our phase generator Table
Spatial Polynomial Gabor Fields. Previously we have discussed the design of the phase gener- ator, which is a 1-D function taking time input and output dynamic phase sequence. We then discuss the design of the polynomial Gabor field which represents the scene spatially.
The major difference between the architecture of temporal polynomial neural fields (T-PNF) and spatial polynomial Gabor fields(S-PGF) lies in the definition of Gj- Specifically, S-PGF is also realized as the summation of different S-PGFs representing different subbands:
F(x) = Σ (x),
(54)
Fx(x) = Σ 9j(x, by: D (x),
(55)
k=1
Zz,(x) = (x, O, A1).
(56)
Zx,((x) (x, O, A&)
(57)
where (x(x, a, b) is subband limited in Ri (a, b, d(0, d(ss), is further defined using Gabor basis:
b,(x,,,b)= = Wxg,(x), (1) W. E Rixd
(58)
where gj is Gabor basis. The hyper-parameters in this model listed in Table 5
17

---

detailed Published as a conference paper at ICLR 2024
Hyper-Parameter Name
Value
Output Dimension
32 (2D Video) / 33 (3D Scene)
Hidden Dimension
64
Number of Bandwidths
4
Table 5: hyyer-parameter used in Spatial Polynomial Gabor Fields.
Layer Type
In Ch
Out Ch
Kernel
Stride
Pad
Notes
Transposed Convolution
32
128
Batch Norm.
Transposed Convolution
256
128
4
Batch Norm.
Transposed Convolution
256
64
Batch Norm.
Transposed Convolution
128
32
4
Batch Norm.
Transposed Convolution
64
16
Batch Norm.
Transposed Convolution
32
32
4
2
Batch Norm.
Convolution
32
32
5
Convolution
32
3
5
2
RGB Output
Table 6: Feature Decoder Architecture.
Basis Modulation and Phase-PGF. To combine the T-PNF and S-PGF discussed above, we in- troduce the procedure of basis modulation and form Phase-PGI represent the dynamic scene.
Intuitively, the dimension of the phase space determined by the number of subbands K. in S-PGE. For each gj E R(())(a (a, b, d(0)), d(H;), d())),8)d, we assign the phase ht)() to it. Therefore, the subband (x(x, a, b) modulated
t, (x e b,, d()),) W Rxxd (59)
Please refer appendix C.2 for a more comprehensive discussion on this topic.
Neural Rendering. In the case of 3D dynamic scene, we further perform neural render- ing (Mildenhall al. 2021) 1) render the 2D feature fields. The first 32 output dimension of Phase- PGF is interpreted as feature fxx) and the last dimension is interpreted as density a(x) used in neural rendering. The 2D feature map m is rendered using the following volume rendering equation:
F(r) =10="" J (()))
(60)
where T(I) exp (-fe, (s)ss)) is the ray being rendered, and d is the view direction.
Feature Decoding and Rendering. We then describe the network architecture that decodes a fea- ture map into RGB rendering. The decoder is bunch of U-Net (Ronneberger al.) 2013) like convolution layers followed by batch normalization. It incorporates skip connections for feature concatenation. Starting with 256 input channels, the decoder progressively reduces and then in- creases the channel size. The final layers produce outputs suitable for RGB images, with Tanh activation function scaling the outputs to specific range. The detailed architecture in Table 6
The Upsampler module employs nearest neighbor upsampling and convolution operations for fea- ture map upscaling. It includes two stages of upsampling, each magnifying the spatial dimensions by factor of 2. Between these stages, convolutional layers adjust the channel dimensions. The configuration outlined in Table
Adversarial Training. During adversarial training, a PatchGAN (Isola et al. 2017) architecture discriminator is used.
18

---

Published as a conference paper at ICLR 2024
Operation
In Ch
Intermediate Ch
Out Ch
Notes
Upsample (x2)
Nearest Neighbor
Convolution
32
128
Kernel: 3, Pad:
ReLU Activation
Upsample (x2)
Nearest Neighbor
Convolution
128
3
Kernel: 3, Pad:
Table 7: Upsampler Architecture.
C.2 DISCUSSION ON THE PHASE SPACE
In this section, we further discuss the initialization and the structure of the phase space. We also perform an experiment to help the reader intuitively understand the phase space.
As previously discussed in appendix C.I the phase space has a dimension K., which is identical to the number of subbands defined in S-PGF. This parameter is further determined by the human prior knowledge of the number motion components in the given scene. However, we argue that our architecture allows redundant number of phase dimensions that be not identical to the motion count in the given scene.
This mechanism of redundant number of phase dimensions is similar Slot-Attention (Locatello) et al. 2020), where there can be empty slots that do not represent an object. In our case, whether a phase represents a part of motion information is determined by the contribution to the final rendering of the subband corresponding to this phase. Mathematically, this is formulated as a phase score sj defined as below:
= Ex,x (x,, (x(x My (x, Ex t)
(61)
(62)
Practically this score is calculated by performing discretization.
We provide an example below demonstrate this argument. As demonstrated in Figure a a), we synthesize video containing ball with the motion of damping vibration. We instantiate our net- work with 16 subbands, each associated with phase. Although such phase space redundant, the space can be analyzed using eq. (62) The result is shown in Figure (7).. from which we can see that phase with index and dominated with score distribution.
We further sample some phases in visualize them in Figure d). For phases having a high score (0 and 2), they align well with the ground truth motion shown in Figure (cc.. From the subband visualization in Figure b)), it can also be seen that they spatially represent the moving ball.
For phases that have low score (4 and 9), they have a relatively chaotic motion. In Figure bb) can be seen that they do not represent meaningful information in the input, indicating that they are "empty phases".
C.3 MANIPULATION OF THE PHASE SPACE
In this section, we discuss the implementation details of the two different types of phase manipula- tions studied in this paper. The result of such manipulation is discussed in Sec. 4.3
Phase Smoothing, The operation of phase smoothing corresponds to "motion smoothing" in Sec. 4.3 Practically, bandwidth upper limit is defined to perform the editing. For given phase sequence h(t) H. the edited phase h is defined follows:
h'(t F (BUB))=F(((((UNN))) (T(F(K(t)) B)),
(63)
TU(B) = [1, Bf
(0. otherwise.
(64)
19

---

Published as a conference paper at ICLR 2024
(a) Frames of the Input Video
(b) Different Subbands of Frame 0. From left to right: 0, 2, 4,
160
0.25
1.0
0.00
Phase
Phase
120
0025
100
6.50
0.75
80
-1.00
100
200
300
400
500
100
200
300
400
500
frame id
(c) Ground Truth Motion Trajectory
(d) Predicted Phases
0.6
0.5
500e
0.1
12
13 14
15
Phase
(e) Bar Chart of the Score of Different Phase
Figure 7: Demonstration having redundant phase space. (a) The input video shows a ball with damping vibration motion. (b) Visualization of different subbands of the reconstructed model, evaluated at t = O. Subbands 0 and 2 (corresponding to the phase and 2) have motion information, while Subbands and (corresponding to the phase 4 and 9) are empty subbands. (c) The ground truth motion trajectory of damping vibration. (d) Some of the predicted phases in the redundant
phase space. Phases and were learned to represent motion information, while Phases 4 and 9 are empty. Please see wttpt:tt///// /
chen www.eeennnnnnnneeeeeeeeeeeee wwwpp//pwppppp.eoeceec..omc #damping for the animation.
where and F-1 are Fourier transform and its inverse transform.
Phase Intensity Adjustment. This operation corresponds to "motion intensity adjustment" in Sec. 4.3
Practically, similar to the phase augmentation procedure described in Sec. Sec. three pa- rameters X, b,, by are defined in this procedure. Given phase sequence h(t) H. the manipulated phase h is defined as:
R MMJA by bb) ((1) + (A - y(((br, b(t bn.) =F b(,b)))
(65)
(66)
where À is intensity manipulation coefficient, by and by subband limits for specific component of the signal, F and F-1 are Fourier transform and its inversion, and T(f) is a band-limin filter defined follows:
20

---

Published as a conference paper at ICLR 2024
(a) Frames of the Input Video
0.66
0.85
50
1.50
250
350
Framme ID
(b) Phase Discovered
Figure 8: Results on Non-Rigid Motion. In (a), we show the sampled frames the input video. In (b), show the discovered phase and associated subband. For animated video, please refer to http://sea//.pppggeeaeeso
[1, ifissfs 0, otherwise.
T(ffi,
(67)
C.4 TRAINING DETAILS
The models trained NVIDIA A5000 GPU. The first stage of training takes around ten hours to converge. The second stage adversarial training takes around three days to fully converge.
D ADDITIONAL EVALUATIONS
In this section, we provide more extensive results of the proposed method to further explore the boundary of the proposed method.
D.1 MORE MACRO MOTION COMPLEXITIES
We first demonstrate several additional motion examples to show that our method can handle varying motion complexities.
Non-Rigid Motion We first show that the proposed pipeline can also deal with non-rigid motion dynamic scene. Specifically, we run the proposed method on an Internet video dubbed Jelly. The results can be seen in Figure 8 hnndhttt:::. chen geng wtt.c/oo///ppceeee comp
can seen that although the motion in this video complex non-rigid, non-regular motion of a jelly, our method successfully discovered its low-dimension phase information by assigning subband to its moving texture feature, as shown in Figure ())
Interaction Between Objects. A more complex motion pattern is when there are interactions between different objects. simulate this case, we synthesize a video called Collision where there moving balls in the scene, and the first ball is collision with the second ball.
21

---

Published as a conference paper at ICLR 2024
(a) Frames of the Input Video
200
Phase
0.075
Prase
0.050
Memmsted Paase
0
125
0.525
0,000
100
-4.050
Frame D
Paam
(b) Ground-Truth Motion
(c) Normalized Phase
Figure 9: Results on Object Interaction. (a), we show the sampled frames of the input video.
We show the ground-trunt motion trajectory in (b). In (c), we show the discovered phases (with normalization). For animated video, please refer to co ission
ccw/phasepggfff
:
:
:
(a) Frames of the Input Video
1.70
160
3.8
150
3.4
830
3.2
2.0
Phone
Phase
Phose
3.00
-02
100
100
20C
200
400
300
Frome
(b) Ground-Truth Motion
(c) Phase Reconstructed
Figure 10: Results on Multi Objects Motion. In (a), we show the sampled frames of the input video. We show the ground-truth motion trajectory in (b). In (c), show the discovered phases. For animated video, please refer to http://cceen.eeeecc.oooppppppsps
geng com http//ppssspppaaaalllll
We show the results this case in Figure 9 and https: http://heenngenggg wwwplengg...m//haaapppeeee collision Our model can successfully decompose the phase space and discover plausible mo- tion patterns: The first ball is moving initially and will change direction later, and the second ball is static at first while will be moving after the collision.
22

---

Published as a conference paper at ICLR 2024
(a) Frames of the Input Video
03
200
180
011
1.0
0.0
120
-02-
80
20
100
(b) Ground-Truth Motion
(c) Phase Reconstructed
Figure 11: Results on Non-Periodic Motion. In (a), we show the sampled frames of the input video. We show the ground-truth motion trajectory in (b). In (c), we show the discovered phases. For animated video, please refer to httts://chen-genett ccmm/plaaseoottorooocc...
Multi-Object Motion. We also study the applicability of the proposed framework in the sce- nario of a slightly more complex motion space. We synthesize a dynamic scene with three dif- ferent moving balls (Ball3 data) with different moving frequencies and trajectories. By using the proposed method to extract the phase information such a scene, we can get the phase space decomposition shown in Figure 10. The animated results can be found at https:
7/chen www........aaeeeeeeeeeeeee pergg,ccm//ppaasspffftt wttp//phasspaaaaall..l
We additionally synthesize three videos, Damping, Bouncing, and Projec- tile, containing different non-periodic motions. In Damping and Bouncing, we simulate damping vibration and bouncing of a ball. In Projectile, we simulate a ball being projected outward, subject- ing gravity.
Non-Periodic Motion
The results in Damping can be found at Figure 7 can be faithfully reconstructed by the extracted phase. Please see httph::t////ht hhtp://cceeeeeeeeee geng.engggg........ com/ phasepg #damping for the animation.
It can be seen that the low-dimensional motion
The Projectile example provides another different pattern of motion. From Figure can be observed that our phase, defined in two dimensions, can be used to decompose the given motion into x-dim and y-dim, representing uniform linear motion and parabolic motion, respectively. The animation can be found at wtthttts:/// /ttps///chenn wwweeenneeeegg..ooheaeeeeeeeeeee http://.......psssssoooooeeeiiia
D.2 QUANTITATIVE METRICS AND EVALUATION
We introduce several metries to quantitatively evaluate the results.
Normalized Cross Correlation. This metric is measure used to quantify the similarity between two signals. For signal [(() and g(),, this metric is defined as below:
Σ,0()) g)
(68)
E.((tt) - ())(t)-f'-,, E,((()) 2
23

---

Published as a conference paper at ICLR 2024
(b) Ours w/o Feat Dec. and Adv. Training
(a) Input
(c) Ours w/o Adv. Training
(d) Ours (Full Model)
Figure 12: Ablation Study. We study the effect of the proposed components by removing them from the full model. In (a), we show the input video. In (b), show the rendering from the model without feature decoding and adversarial training. In (c), we show the rendering from the model without adversarial training. (d) shows the rendering of our full model.
We calculate this metric between the predicted phase and the ground truth motion trajectory in two synthetic data: Projectile and Damping. The results are shown in Table Compared the baseline, our model extracts phases that are more aligned to the input trajectory.
Fréchet Inception Distance (FID). We calculate the FID score (Heusel et al., 2017) between the input video and the manipulated video evaluate the rendering quality of different methods. We perform the evaluation on a real captured video Airpods. The video can be found at https: 7/chen-geng. com/ phasepg
The result can be found at the third row of Table Our method surpasses all the baselines signifi- cantly in terms of rendering quality.
D.3 ABLATION STUDIES
We conduct ablation studies on the proposed components to validate the effectiveness of the pro- posed method. The results can be found at Figure
The Effect of Feature Decoder It can be seen from Figure 12 and (c) that when we remove the feature decoding module, the rendering quality has degraded by large magnitude. The reason that only using S-PGF will produce low-resolution images. By using feature decoder, we can render higher-resolution images.
The Effect of Adversarial Training By comparing Figure 12 and (d), we can see that adding adversarial training lets the model learn more information on the detailed texture of the object.
D.4 MORE ANIMATED RESULTS
We refer the reader to the supplementary website: https::/chen--00 com www.engg..om/haaspgg.... pwwsepgfasppppppppp for more animated results.
D.5 DISCUSSION ON THE POINT TRACKING METHODS
Another possible method to perform macro motion analysis is by doing dense point tracking method and figuring out some method to extract sparse motion information from the dense tracked particle trajectory. However, tained particle trajectories of high dimension, which not be easily analyzed.
non-trivial to perform this process. After point tracking, the ob-
We make an attempt in this section to use a State-of-the-AAe point tracking method PIPs+ (Zheng et al. 2023) together with dimension reduction technique get sparse motion feature sequence. Specifically, we sample dense tracked particles using a grid strategy following (Zheng et al. 2023) and perform dense point tracking. This results in high-dimensional motion trajectory. We then
24

---

Published as a conference paper at ICLR 2024
150
Component Component
100
Component
50
-50
-100
50
100
150
200
250
Frame
Figure 13: Two major components from the point tracking baseline. The phase generated by our method can be found at Figure See visualization of tracked particles at/ht chnn- htmm/pphasepf//trckinnn html
Key Frame 1
Key Frame 2
Key Frame 3
Input
Subband 1
Subband 2
Figure 14: Moving object discovery on Giraffe data. By running the proposed method on the data, the moving object and static objects separated into different subbands.
perform Principal Component Analysis (PCA) on the obtained dense motion to extract its dominant dimensions get sparse motion trajectory. The animated result can be found https:// html
chen-geng.com com/ ott.erreeeeg..ooooopppaaeee/e//e
We visualize the extracted motion components in Figure It be that the extracted low- dimensional motion representation noisy and is not plausible for the motion macro motion analysis task in this paper (Cf. the phase generated by the proposed method in Figure 3
25

---

Published as a conference paper at ICLR 2024
0.92
0.90
0.45
1373
1320
1320
25
(a) Input Video
(b) Subband
(c) Phase
Figure 15: Failure cases. Animation: http:r//eaaaageegg...oooo/pppppp
D.6 DISCUSSION ON MOVING OBJECT DISCOVERY
The proposed method can be potentially used for finding moving objects in a dynamic scene. In this section, we make a preliminary attempt at this.
We capture a video called Giraffe and run our method on this video. The result can be found in Figure [4] In this example, the moving red ball and the static giraffe toy are separated into two different subbands. Using this decomposition, we can detect the moving objects in dynamic scene.
D.7 FAILURE CASES
In this section, we provide some failure cases of the proposed method to help the reader better under- stand the boundaries and limitations of the proposed method. The results can be found in Figure [5] and please refer to https: htaps://chenns http///hhennnnnn..cee geng. hwtp:/aseee/..taaaaaaacaaeeeeeee lure for animation in this section.
Chaotic Motion Hair For chaotic motion such as hair blowing, there is no underlying low- dimensional representation the scene motion. this case, the method can only discover coarse motion tendency yet cannot recover the full motion space.
Motion of Shapeless Objects Fire The proposed method assumes the objects in the scene have a concrete shape. For data like fire where the objects shapeless, method can only discover the very coarse moving part, but cannot clearly separate different motion parts.
26


Reviews:
[{"review_id": "F13LFoh08N", "reviewer": "Reviewer_EQJi", "summary": "This paper delves into the exploration of motion analysis through the use of phase-based representation, with an emphasis on large-scale, rigid-body movements, also known as macro motion. In contrast to localized and small-scale motion, macro motion operates on a broader scale, encompassing extensive spatial and temporal transitions that are easily perceptible by humans. Traditional methods often fall short in accurately capturing such features because the filter operations are primarily designed to handle low-level features.\n\nTo overcome this limitation, this paper introduces a novel approach that utilizes phase-oriented neural Gabor fields as input embedding to reconstruct the original images or 3D scenes. By a reverse optimization process, the optimized phase input can encode some useful information while keeping the periodic. As the fundamental element in the real world, some good properties of phase can be also useful in further analysis. To enhance this reconstruction and reverse optimization, various training strategies like multi-stage, deep latent, and adversarial training have been proposed. Despite the lack of extensive evaluation, these methods are expected to deliver promising results.\n\nIn the experimental section, through some examples, it shows the method can outperform some baseline methods in extracting more reliable alignment. Moreover, the properties of the phase support the smoothing and separation of motions, which can be beneficial in specific applications like magnification.", "questions": "Where is the boundary of the phase space this method can support? There can be highly varied frequencies in the real world, but the working range of the proposed method is unclear. It would be useful to explore these boundaries by conducting experiments with various artificial videos.\n\nHow to determine the number of phases? In some cases, there appears to be one phase, while in others there are multiple. Is it possible to use an excessive number of phases to overfit a scene? Conversely, what would be the outcome if we used a single phase for a scene with multiple objects?\n\nThe extracted phases seem to care about the frequency more compared to the amplitude variations. In the second video of Fig.2, different balls do have different spatial transitions, but they seem to share the same peak. \n\nThere is an option to use point tracking to extract the phase from the tracking path. How does this method complement them, and in what ways does it offer unique advantages compared to other methods?\n\nAre the extracted phases open to manipulation? For example, if we multiply the phase by a coefficient A, will the corresponding recovered image or video be amplified? It is claimed in the paper that we can manipulate, but the results seem to be missing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The story is clear, the reader can easily understand the importance of the phase and the effectiveness of the proposed method.\n\nThe idea is interesting and novel. Using the phase as the fundamental feature to describe the motion can help to understand the motion better. \n\nI do like the inverse optimization with the Gabor field for locating the best-aligned phase feature; the theoretical exposition is clear, straightforward, and logical; and the proofs are also clear. \n\nThe employed coarse-to-fine generation framework and strategy effectively facilitate the learning of priors.\nThe results provide some evidence to support the central thesis of the paper.", "weaknesses": "While I appreciate the innovative concept presented in this paper, I believe the current version exhibits some weaknesses that need to be addressed:\n\nSome of the exposition needs to be improved. Some details and properties are not clearly clarified. Some points need more in-depth discussion:\n\n1. The input phase space and the details of the Phase Generator are not elaborated. As the central element in this model, it’s not clear how to initialize the number of phases, and how the current framework adapts to the diverse scene. The full phase space can be much more complex than that in those demonstrated examples. Hence the potential of this method is unclear.\n\n2. The manipulation of the phase has been mentioned a few times, but it hasn't been detailed and demonstrated further in the paper. If the phase space is manipulatable, it will be important in the model interpretability. \n\n3. The implementation of using Phase-PGF in the deep neural network is not described clearly enough. \n\t\nThe contribution is unclear with insufficient evaluation. A more comprehensive evaluation is needed to demonstrate the boundaries of working examples, and provide more insights:\n1. Quantitative evaluation: Some of the test videos are artificial. In this case, we may want to synthesize more videos that contain varying motion complexities, for a more extensive evaluation.\n2. Ablation study: An ablation study for each network module would be beneficial.\n3. Evaluation metrics: A metric to measure if the phase meets the requirements would be useful.\n\nLast, some existing methods use different formulations to perform similar motion analysis, it is strongly recommended to add the comparison with them. For example, some point tracking methods can also extract rigid body movements as sparse points, and using these predictions it's also possible to extract interpretable features of the motion, such as phase or other high-level descriptions. While I don't question the novelty of this paper as it employs a *unique* methodology, it would be beneficial to add more experiments and discussions on these approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofGbE5DnUv", "reviewer": "Reviewer_no8y", "summary": "The paper proposes to represent scene dynamics with low-dimensional time-varying phases. The representation is learned using Phase-based neural polynomial Gabor fields (Phase-PGF). The paper argues that Phase-PGF has several properties which make it suitable for motion analysis task like motion loop detection, motion factorization,  and motion magnification.", "questions": "- sometimes it is hard to say which phase is right. Maybe participants are just biased towards more periodic-looking signal even if it does not really match the scene motion. I wonder if there is more systematic way to evaluate this? For example, we can detect the periodicity of a given motion and then compare its frequency with the predicted frequency.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses an interesting application and is nicely written and easy to read.  \n- The paper proposes a novel formulation (Phase-PGF) for scene motion.\n- Phase-PGF several properties which make it suitable for motion analysis. Two properties I found interesting are :Periodicity correlation and motion separation.\n- The paper show several interesting motion analysis applications like motion separation, motion factorization, motion magnifications\n- The results shown seems to surpass previous work", "weaknesses": "- Although the method is novel, I think the supporting experiments are insufficient to validate the method.\n- Not enough results in the supplementary webpage. All the results, except one, are visualizing the phase. Only one result show a generated video. This is insufficient to judge the quality of the results. I would have liked to see a video of the motion separation experiment and more videos for the motion intensity adjustment. \n- Motion editing results display visual artifact\n- Most of the experiments are on toy examples with simple objects or simple motion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "goKlF8hNu2", "reviewer": "Reviewer_WSr6", "summary": "This paper formulates and studies a new task of macro motion analysis. To perform macro motion analysis, the authors propose to learn an implicit function with respect to point coordinate x and time t, composed of Gabor functions and phase functions, namely Phase-PGF. By studying and adjusting the phase function of the Phase-PGF, one can infer periodic motion detection, motion separation, motion smoothing and motion intensity adjustment. Additionally, in order for the Phase-PGF to be able to extrapolate to unseen motions, which is needed for motion intensity adjustment, a discriminator on decoded images is used for adversarial training. Experiments on 2D video and 3D dynamic scenes are performed. Both human preference and visual results show its effectiveness and correctness on several motion analyses tasks.", "questions": "1. Currently, the extracted motion phase is agnostic of the motion direction. Is there a way to decompose motion phases, for example, to x and y directions.\n2. How, in practice, are the motion representation, e.g., Figure. 2(a), extracted from learned implicit function?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, a new problem of macro motion analysis is properly formulated and approached with reasonable method, proven effective by adequate experiments.\n2. Using phase functions to model dynamic scenes has advantages for motion analysis compared with other dynamic scene representations.\n3. I appreciate authors experimenting with some real-captured data and the efforts of increasing image rendering quality.", "weaknesses": "1. It would be better to design some automatic metrics other than human preference. For example, trackers can be put on moving objects to obtain ground truth motions, which can be used to compare with predicted motions. For the motion intensity adjustment experiment, to evaluate the visual quality, some metrics designed for generation tasks, e.g., FID, KID, can be used.\n2. Currently, the examples in paper and supplementary website show very simple movements, mostly periodic. I wonder how would the method apply to more complex, dynamic scenes. Not necessarily dynamic 3D scenes, more complex 2D video can also do the work. If so, can the method separate more than two motions in the scene? And can some algorithm be designed to automatically find the moving object?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WYvmxpUbmh", "reviewer": "Reviewer_NjSx", "summary": "This paper focuses on macro motion analysis, which involves collecting all visually observable motions in dynamic scenes. The traditional filtering-based methods typically pay attention to local and tiny motions, while recent dynamic neural representations can represent motions faithfully, but cannot be directly applied to motion analysis. The authors propose Phase-based neural polynomial Gabor fields (Phase-PGF) to represent scene dynamics with low-dimensional time-varying phases. Phase-PGF has several properties suitable for macro motion analysis, and it can be used for various dynamic scene editing tasks, such as motion loop detection, motion separation, motion smoothing, and motion magnification. It further implements the Phase-PGF using an innovative neural architecture and a refined training approach to enhance the quality of dynamic scene representation and editing.\n\nThe main contributions of this paper are as follows:\n1) formulate the macro motion analysis problem \n2) provides a novel phase-based neural polynomial Gabor fields (Phase-PGF) approach to tackle the motion analysis problem\n3) demonstrates the effectiveness of the Phase-PGF approach on both 2D and 3D scenes\n\n---------------------------------------------------------------------------------------------------------------------------------------\nAfter reviewed the author's rebuttal, I have increased my score.", "questions": "1. In the related work section, the authors discuss concurrent works on tiny motion editing. How does Phase-PGF differ from these methods in terms of addressing macro motion analysis?\n\n2. The authors mention that Phase-PGF may present slight artifacts when magnifying large motions. Could they provide further insights on the reasons for these artifacts and potential solutions to address this issue?\n\n3. Are there plans to improve Phase-PGF's scalability for large-scale 3D dynamic scenes?\n\n4. In the experimental section, it would be helpful to know more about the datasets used and the evaluation metrics employed for assessing the performance of Phase-PGF. For example, for the human preference study, how many ratings are obtained for each video, and how many videos are being used in the study? \n\n5. Can the authors discuss the potential applications of Phase-PGF beyond the mentioned motion analysis and editing tasks, such as object tracking, motion generation, or other real-world scenarios?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strengths of this paper are as follows:\n1. Originality: The paper proposes a novel approach called Phase-based neural polynomial Gabor fields (Phase-PGF) for representing and analyzing dynamic scenes. This approach addresses the gap in the field of computer vision, where traditional methods focus on local and tiny motions, while recent dynamic neural representations lack direct applicability for macro motion analysis. The paper's originality lies in its focus on macro motion analysis and the introduction of Phase-PGF as a suitable representation for this purpose.\n2. Quality: The paper presents a detailed analysis of the proposed approach, discussing its theoretical properties and demonstrating its ability to handle various macro motion analysis tasks such as motion separation, motion smoothing, and motion magnification. The experiments conducted showcase the effectiveness of Phase-PGF in representing and editing dynamic scenes, making the paper's contributions of high quality.\n3. Clarity: The paper is well-structured and clear. The authors formulate the problem of macro motion analysis, explain the theoretical properties of Phase-PGF, and discuss the implementation and training of the approach. The experiments are described in sufficient detail, making it easy for readers to understand and replicate the proposed method.\n4. Significance: The paper addresses a key challenge in the field of computer vision, motion analysis, and dynamic scene representation. By focusing on macro motion analysis, the proposed approach has the potential to impact various applications, such as motion tracking, generation, virtual reality, and other computer vision tasks. The paper's contributions are significant in providing a novel solution to representing and analyzing large motions and 3D scenes, which has been a largely underexplored area in the field.\n\nIn summary, the main strengths of the paper are its originality, quality, clarity, and significance. The paper presents a novel and effective approach to representing and analyzing macro motions in videos (both 2D and 3D), discusses its theoretical properties, and demonstrates its practical applicability through experiments, making it a valuable contribution to computer vision and related domains.", "weaknesses": "Weaknesses of the paper:\n1. Slight artifacts in boldly magnifying large motions: The paper acknowledges that when magnifying large motions, there are slight artifacts in Phase-PGF. The cause of these artifacts may stem from the neural network architecture or the way motion is represented. To reduce these artifacts, additional investigation and optimization of the model may be necessary. \n \n2. Limited scalability to complex large-scale 3D dynamic scenes: The paper acknowledges that Phase-PGF might not perform well in complex large-scale 3D scenes due to computational efficiency issues. As the paper's focus is on macro motion analysis, addressing this issue is crucial to improving the applicability of the proposed method, especially for complex scenes. A possible solution could be the use of a spatially adaptive Gabor basis as mentioned in the paper. \n\n3. Insufficient experimental evaluation: Although the paper presents some experimental results, it would be beneficial to include more comprehensive evaluations with various datasets and tasks. For example, additional experiments could be conducted on larger and more diverse datasets, varying scene conditions, and different motion types (e.g. human motion). This would help establish the robustness and generalizability of the proposed method.\n\n4. Neural network architecture: The paper uses a neural network to instantiate the Phase-PGF representation. However, the choice of the neural network architecture could be further optimized for the specific task of macro motion analysis. Including the details of the architecture may help to improve the reproducibility of the proposed work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)