Self-Supervised Motion Magnification by Backpropagating Through Optical Flow
Authors: Zhaoying Pan
Daniel Geng
Andrew Owens
Venue: NeurIPS 2023 poster (2023)
Keywords: Video Processing
Motion Processing
Motion Magnification
Optical Flow
TL;DR: We magnify motions in videos by differentiating through optical flow, allowing for a self-supervised magification algorithm

Abstract: This paper presents a simple, self-supervised method for magnifying subtle motions in video: given an input video and a magnification factor, we manipulate the video such that its new optical flow is scaled by the desired amount. To train our model, we propose a loss function that estimates the optical flow of the generated video and penalizes how far if deviates from the given magnification factor. Thus, training involves differentiating through a pretrained optical flow network. Since our model is self-supervised, we can further improve its performance through test-time adaptation, by finetuning it on the input video. It can also be easily extended to magnify the motions of only user-selected objects. Our approach avoids the need for synthetic magnification datasets that have been used to train prior learning-based approaches. Instead, it leverages the existing capabilities of off-the-shelf motion estimators. We demonstrate the effectiveness of our method through evaluations of both visual quality and quantitative metrics on a range of real-world and synthetic videos, and we show our method works for both supervised and unsupervised optical flow methods.

Self-Supervised Motion Magnification by Backpropagating Through Optical Flow
Zhaoying Pan"
Daniel Geng" University of Michigan
Andrew Owens
httpa://dangeng http://dannegggiihhbb... io/FlouMag
Abstract
This paper presents simple, self- sepf-suue method for magnifying subtle motions in video: given an input video and a magnification factor, we manipulate the video such that its new optical flow is scaled by the desired amount. To train model, we propose a loss function that estimates the optical flow of the generated video and penalizes how far if deviates from the given magnification factor. Thus, training involves differentiating through a pretrained optical flow network. Since our model is self-supervised, we can further improve its performance through test-time adaptation, by finetuning it on the input video. It can also be easily extended to magnify the motions of only user-selected objects. Our approach avoids the need for synthetic magnification datasets that have been used to train prior learning-basee approaches. Instead, leverages the existing capabilities of off-the-shelf motion estimators. We demonstrate the effectiveness of our method through evaluations of both visual quality and quantitative metrics on a range of real-world and synthetic videos, and we show our method works for both supervised and unsupervised optical flow methods.
Introduction
Motion magnification methods [32, 58, 40] increase the size of tiny motions in a video, revealing subtle details that are difficult to discern with the naked eye. However, existing methods come with significant limitations. Early hand-crafted methods generally require small periodic motions [64, 58] or human in the loop [32]. More recent supervised learning methods [40] require ground-truth training examples, such as videos before and after magnification, which are difficult to obtain without synthetic data. Creating this synthetic data is a challenging problem, since it seemingly requires capturing all of the possible objects and motions that one might ever want to magnify.
In parallel, the field of motion estimation has addressed many closely related challenges. Modern optical flow networks [65, 54, 51, 10] are designed to track objects undergoing complex motions, both large and small. Most of these methods are trained on supervised datasets that capture wide variety of objects, but parallel work has shown the effectiveness of unsupervised flow estimation [33, 27]. We ask whether we of their existing capabilities and reducing the need for
motion estimation models to train magnification models, taking advantage training data.
We propose a simple motion magnification model whose supervision signal comes from an off- the-shelf motion estimation model. Our method exploits the fact that optical flow networks are differentiable, and thus can be used as part of a loss function. We train model to take pair of video frames and a magnification factor a as input, and to generate a new pair whose predicted optical flow is a times as large as that of the input. We simultaneously optimize a regularization loss that preserves the visual appearance of each tracked pixel in the generated video. Notably, our model can
"Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(1) Original Frame
(6) No-TTA Frame
(1) Original Frame
(9) Left Fork
(iF) Right Fork
Original
@ Original
()) Orggina
(v) Orignnal
Magnilynn Left Fork
No
(5) Magnified
(1) Magnrfaad
(ii) TTA Frame
[v] TTA
(V) No Targating
(vi) Magnifynng Fork
(a) Magnification
(b) Test-time Adaptation
(c) Targeted Magnification
Figure 1: Magnifying Motions in Video. (a) We show frames along with closeups (Left) or y-t slices (Right) from original videos and magnified videos. The spatial location of the y-l slice is visualized by superimposing colored line the still frames, and the location of closeups is shown with rectangle colored green. (b) We show the frames from the original video, the magnified frame NTA),, the magnified frame after test-time adaptation (TTA), and the y-t slices of the three videos. Test-time adaptation (i.c., finctuning on the input video) improves generation quality as can be seen in the y-t slices. (c) We magnify the motions within a user-selected object segmentation [29] and show y-t slices of the magnified vidcos on different targets. We red/green colored lines to indicate the locations of g-t slices for different targets.
be trained solely using real unlabeled videos. The optical flow estimator only used within the loss function during training-ww we do not require it at test time.
Our model's 's simplicity and its ability to be trained solely through self-supervision provides it with several ad advantages over other approaches. We show that we can improve our generated videos through test-time adaptation [52, 26] by finetuning on a given input video. Our formulation can also be easily extended to magnify the motions of a single object, specified via a user- providedd mask. In addition, we show results using both off-the-sheii supervised [54] and unsupervised [33] optical flow methods.
Our method is closely related to Lagrangian magnification methods [32, 19], which tracks individual pixels through a video and then amplifies their motion. While this approach is intuitively appealing and the basis for the earliest magnification methods, major shortcoming is that it is not clear how to combine the tracking and synthesis together. Previous Lagrangian approaches [32] often rely on hand-crafted pipelines that combine motion estimation, warping, and inpainting steps. By contrast, our method avoids these pitfalls by using off-the-shell image-to-image translation architectures [45], and uses tracking only within the loss function.
We demonstrate the effectiveness and flexibility of our model in several ways. First, we show through experiments real and synthetic videos that the optical flow in the generated videos more closely matches the desired motion than videos generated by baseline approaches, such as methods based on warping or supervised learning [40]. Second, we show qualitatively that model can successfully magnify motions for a variety of videos, containing both small and large motions. Finally, we show that we can improve generation quality using test-time adaptation, and magnify individual objects using user-supplied segmentation maps.
2 Related Work
Lagrangian magnification. Motion magnification methods can be broadly categorized as either Lagrangian [32] or Eulerian [64], a classification borrowed from fluid dynamics. Lagrangian methods explicitly track pixels, then generate new frames by forward warping (i.c., splatting) using magnified velocity estimates. Liu et al. [32] originally proposed the motion magnification problem and solved it using Lagrangian approach. They modeled motion as feature trajectories and (with human- in-the-loop assistance) cluster pixels by similarity in position, color, and motion. Recent methods have extended Lagrangian magnification with more accurate optical flow [17]. While our approach performs explicit tracking and thus closely related to Lagrangian methods, we decouple generation from tracking: we produce images using off-the-shelf networks and track only within the loss function. This allows us to avoid some the challenges of the "warp and inpaint" approach, such as handling occlusions and filling holes.

---

Eulerian magnification. Eulerian methods magnify motions without explicit motion estimation. Instead, they generate a magnified video by amplifing the temporal changes at fixed locations/pixels. Wu ef al. [64] decomposed video into frequency bands and applied temporal filters to extract a signal at a specific bandpass. The extracted signal then amplified by a magnification factor and added back to the video. Wadhwa et al. [58] proposed a phase-based method, using complex steerable pyramids [47] to decompose the video and separate the phase from the amplitude to amplity phases. Later work improves efficiency using Riesz pyramids [59] and removes large motion by decomposing a scene into foreground and background layers [1]]. Zhang er al. [72] magnified small motions while ignoring large motions by amplifying the motion field acceleration using second-orde temporal filters. Concurrent work [15] extended 2D Eulerian methods to 3D motion magnification. Eulerian methods are well-suited to tracking small motions at high spatial frequencies, and may struggle when handling large motions [58], whereas Lagrangian methods' magnification quality is determined by the quality of the optical flow predictions [64]. Since our method's loss function is defined using optical flow, its capabilities are more similar to those of Lagrangian methods.
the
Learning-based magnification. Several recent works have proposed learning- motion mag- nification models. Oh et al. [40] created synthetic dataset containing image segments extracted from PASCAL VOC [12] as moving foreground and images from COCO [30] as background, and trained model to regress a ground truth magnified image from two video frames. Inspired by the steerable pyramid used in Eulerian methods [58], they also proposed an architecture that has inductive biases that encourage it to generate crisp images akin to those in Eulerian magnification, by decomposing the image into shape and texture representation. Other work has extended this supervised magnification approach using 3D CNNs [28] and lightweight architectures [48], and has magnified microexpressions using attention-based models [63]. In contrast to these approaches, our method is self self-supervised and learns from unlabeled videos. Recently, Gao et al. [19] use a very similar supervised learning objective and dataset but augment the model of Oh et al. [40] with inductive biases that encourage Lagrangian- like magnification by adding an attention map that is guided by an optical flow field. Instead of making optical flow part of our architecture, as inductive bias that aids supervised training, we use it to define a self-supervised loss function.
Motion estimation. The field of motion estimation is closely related to motion magnification. Early work solved linearized models after making color constancy assumptions [35] or solved smoothhness-regullrizeed models with variational methods [23]. Later work added robust losses and inference strategies [49, 3, 31, 14]. While these methods are well-suited to the subtle motions considered in motion magnification, it is difficult to use them part of a loss function (as do in this work), since it is not straightforward to differentiate through during gradient- based learning. Recent methods based highly accurate. A variety of recent methods train neural networks using real or synthetic optical flow data [16, 25, 43, 24, 67, 51, 65] or unlabeled video data [71, 44, 61, 34, 27, 2]. Our use of pretrained supervised motion magnification datasets.
neural networks address this issuc, since they are differentiable and
flow model removes the need for
Optical flow within a loss function. We take inspiration from recent work that uses differentiable optical flow models to define loss functions that internally perform motion reasoning. Geng er al. [20] used flow to obtain robustness to small positional errors on image generation tasks. Goyal et al. [21] used flow to measure the distance to a goal state for robotic planning. Other work has backpropagated through optical flow as part of a pose estimation model [55]. By contrast, we use flow to compare the motions in two videos.
3
Method
We describe traditional Lagrangian motion magnification, and our proposed self-supervised La- grangian magnification model.
3.1 Lagrangian Motion Magnification Overview
Lagrangian motion magnification methods magnify motion by tracking pixels over time, then resyn- thesizing the video such that the motion of each pixel has increased by a desired amount. More concretely, a point x in the initial frame Lo of video might be displaced by motion field T(x; t) in
3

---

F(lo,
F(lo, 1i)
Lmag
To
1g
Lentor
f
P
correspondences
from JF
(a) Architecture
(b) Losses
Figure 2: Motion Magnification Model. Given reference frame Jo. frame to be magnified It, and map of per-pixel magnification factors a, we predict magnified frame I, We minimize two losses that each use an off off-the the shel- optical flow estimator F(.,)). First, flow of the generated video to be times large that of the input video. And second, also include consistency loss Laske that measures the visual similarity of corresponding pixels in 1 and I,
magnification loss Lung that encourages the optical
frame t. Assuming color constancy, we have:
1x(x T(x;()) [x(x),
where is the frame time and T(.) indicates pixel access. To generate version of the video magnified by factor a', one could synthesize a new frame by increasing the distance traversed by each pixel:
[x + aT t)) Io(x).
(2)
For example, early work in Lagrangian motion magnification [32] aimed to achieve this by performing forward warping (splatting). However, this leads holes and aliasing artifacts [53]. Moreover, there no mechanism for dealing with occlusions: after warping, background pixels may move in front of
the foreground. Finally, such model would not be able deal with appearance changes without additional constraints (c.g., due to lighting variation).
3.2 Self-Supervised Lagrangian Magnification
We propose motion magnification model that avoids the limitations of previous Lagrangian methods, and that can be trained solely using unlabeled video (Figure 2). Consider F(J,, It). the motion field obtained by computing optical flow between reference frame To and another frame I,. Our goal is to generate magnified frame ī, such that its predicted flow is aF It), where the magnification factor. To do this, we optimize an objective to minimize the difference between the estimated flow F(T. Ie) and the desired flow:
Lymag = ||oF (.) - F(J,, 1,),,,
(3)
which term the magnification loss. When F is implemented neural network, optimize this loss using gradient-basee basedd learning methods.
Optical flow models are invariant to variety of photometric changes (e.g., changes in illumination). Thus, optimizing Eq. alone would result in an underconstrained problem. To ensure that the generated frames match the colors in the input video and to also regularize our problem, use an additional color loss. This loss ensures that corresponding pixels in generated frame I, and original frame I are the same color. To enforce this, put both frame and I, into correspondence with the common reference frame la by backward warping each frame with their respective flows: F(lo, 4x) and F(o, It). We then measure the distance between the warped frames. This results loss:
Leoler ||warp F(o. I)) - warp(ī, F(Jo. 12))22
Note that some previous works 164, 58, 40J define the magnification factor in way that is equivalent to (instead of a), which is more conducive their theoretical analysis.

---

This loss also disincentivizes adversarial examples against the flow network. Our full loss is weighted sum of these two losses:
C L Axolor
One benefit of putting optical flow into the loss function that our model needs access to the flow network only during training: inference time the model simply takes in frames and outputs magnified frame. Pseudocode for our method for training and inference can be found in Algorithm and Algorithm 2 respectively.
3.3 Image Generation Architecture
One benefit of our formulation is that our proposed loss (Eq. 5) independent of the image generation architecture, that "off-the-shelf"" image translation architectures can be used to create a magnified im- age. This is in contrast other learning- maannngg-bssddd meth- ods, which incorporate inductive biases within the network to generate images that closely resemble the input ge.g..uo by altering the structure an image while preserving the texture [40]). To generate magnified frames, we use standard U-Net architecture [45] that takes two input frames (the reference frame and the frame to be magnified) concatenated channel-wise. We encode magnification factor a using a sinusoidal positional embedding [39] and tile it to match the same spatial dimensions as the input frames. We con- catenate this embedding channel-wise to the input frames. Please see Section A2 the appendix for full architectural details.
Algorithm Pseudocode a Py PYTorch-like style for training U-Net for motion magnification.
Load data of tuo-frane videos
for (im0, imi) in dataloader:
Sample alpha get enbadding semple
pe_a aacle__appaaae (min,aaainnaaa) (nin_ alpha, nax_aIpha) boreeioaaaoobbeddingee.. pe_a sparial_ttieepe...
a Predict nagnified frane with network input concat((im), im1, pe_a]) imi_nag (Net(intut)
Estimate the F. E_ere aptical_flov(ing inf) P_tgt fIni,a, imi_mag)
Warp the second freme varp_ iml warp(imi, F_erc) varp_imi_mag varp(inl_mag, F_tgt)
à Calculate losses mag_loss 11_loss(a F_arc, F_tgt)
color_loss - 11_1oss (xarp__ farp_ini_nag)
less = mag_loss + veight color_loas
backvard))
3.4 Targeted Magnification
optinizar.atepl
Our model gives us the ability to vary the magnifi- cation factor spatially within an image. We achieve this by providing different values of a for each pixel inference time. This works even when training with spatially constant alpha maps, due to the fully convolutional nature of our network2. As a special case, we magnify the motion of single object by setting the magnification factor to value of a within an object segment, and to I everywhere else. In practice, we use the recent Segment Anything Model (SAM) [29] to extract mask for given object in the reference frame and then dilate the segmentation mask by small fixed number of pixels. We pro- vide pseudocode for targeted magnification during inference in Algorithm 2.
Algorithm Pseudocode in a Porchh-liic style for inference (targeted magnification).
ind video magnified_vider [im0]
franes from input video
for frane in video[1:]:
Get masked enbedding for input alpha pe_a positional_ enbbdddddna)))ddddg((( pe_a apatiel_tileepe._ nack pe_a
Predict magnified frane with a netvork input concat ((mo,, frane, pe_a]) mag
(input) magnified_videe widiedddvvieeeeeeeeeeeeeeeeeeeed www.diddddd.ddddaagggn....oo (pag.rraa))
3.5
Test-Tima Adaptation
Since our model is entirely self-supervised, can finetune it at inference time on the input video [52, 26, 18]. This allows us to adapt to new motions and content. We find that this can significantly improve results, especially for out-of-domain videos. Previous supervised approaches do not have this capability as they require labeled data. To perform rest-time adaptation we take the frames in a given inference video, apply minor cropping and color augmentations, and finetune the model with our loss.
2 While the model could be trained with random spatially-varying a values, we did not see qualitative improvement when doing this.
5

---

Experiments
We evaluate our model both quantitatively through experiments with real and synthetic data, and qualitatively on real videos. We highly encourage the reader to view our website, since the magnified motions can be challenging to visualize in static images.
4.1 Implementation Details
We train two variations of our model. One that uses ARFlow [33], and one that uses RAFT [54] as the optical flow network. Since ARFlow is self-supervised, using it results in a fully self-supervised motion magnification method. RAFT the other hand is supervised flow model with very good performance, giving us a weakly-supervised magnification method that serves as a rough upper bound to the performance of a fully self-supervised method. More discussion can be found in Section A2 in the appendix. All qualitative results presented in this paper are from the RAFT model, except in Figure in which we compare the ARFlow and RAFT models.
Because motion magnification is multiplicative, we sample random magnification factors a exponen- tially, such that logz(a) ~ U (logg(Amin) (O) )). where mmin = 1 and Okmax 16. For larger magnification factors that exceed Omax we find that recursive application of our model produces high-quality predictions. More implementation details are available in Section A2 in the appendix.
4.2 Dataset
Because our method is self-supervised, benefit from large, diverse dataset of videos. To this end we curate a dataset containing 145k unlabeled frame pairs from several existing datasets, including YouTube- YouTube-VOS-2011 VOS-2019 [68]. DAVIS [42], Vimeo-90k [66]. Tracking Any Object (TAO) [8], and Unidentified Video Objects (UVO) [60]. We remove frame pairs with large motions (e.g., from the camera objects), since these frames are less likely be used in magnification appliactions. We also remove frame pairs that are near-identical by setting a lower bound on the MSE between the two frames. addition
training set, we collect a test set consisting 650 frame pairs for evaluation. which we refer to as real-world test set. We provide more details of dataset collection and filtering in Section A3 of the appendix.
For testing, we also use the synthetic test set from Oh et al. [40]. which is generated by compositing objects from PASCAL VOC [12] to backgrounds from COCO [30] at varying levels of subpixel motion and noise. We refer to this test set as the synthetic test set. Finally, we qualitatively assess our method and baselines on various real-world videos, which vary greatly in subject matter, motion complexity, and lighting conditions. These videos include de-facto standard benchmarks for motion magnification used in previous works [64, 58, 59, 5, 40], well as new videos.
4.3 Metrics and Baselines
Metrics. Similar to Oh et al. [40], we use SSIM [62] an evaluation metric. However, because SSIM requires ground-truth magnified images, we can only use this metric to evaluate models on the synthetic test set. In order to evaluate our method on the real-world test set, we propose another metric for motion magnification that does not require ground truth, which we term motion error. Inspired by the accuracy and robustness of recent optical flow methods, we assume that the flow estimate from an optical flow method is ground truth and calculate an end-point error between the predicted flow and the desired magnified flow.
This metric is identical to mag if the same optical flow model is used for evaluation as is used during training. In order ensure robust metric, calculate the motion error metric using wide range of optical flow networks, including PWC-Net, GMFlow, and RAFT, that have been trained on various datasets. We set the number of iterations for RAFT 20 during evaluation, while keeping it during training. This serves the dual purpose of allowing us to train more efficiently, but also evaluate more equitably.
In addition, we compute the per-pixel ratio between the flow magnitudes of the unmagnified and magnified frames, and calculate their average deviation from the desired magnification factor This metric, which we refer to as magnification error, was not explicitly trained for and serves as another indicator of magnification quality.

---

(i) Original
(iv) et
Original
(M)
H Original
(iv)
WWW
(i) Warp
(v) Ours
(i) Warp Nearest
(V) Ours
(i) Warp
(V)
M
(ii) Warp Bilinear
(vi) Ours (TTA)
[ii] Warp Bilinear
(vi Ours (TTA)
(ii) Warp Bilinear
(vi Ours (TTA)
(a) Baby
(b) Cats
(c) Pole
Figure 3: Qualitative Comparison. (Top) visualize the results of various models by plotting y-t slices through vidco volumes. (Bottom) We also show the reference frame and the locations of the y-t slices. Oh er al. (iv) shows noticeable artifacts, indicated by red circles. In (vi) we see that test time adaptation (TTA) can improve our method on out-of-domain inference videos.
Oh et al.
Ours
(Train)
Input (Boiler)
Oh
Input (Camel)
Oh et Ours
Figure 4: Image Quality Comparison. We show magnified frames generated from the method of Oh ef al. and method. Because the model of Oh al. is trained synthetic dataset, it may generalize to the presence novel motions such as camel chewing, or novel scenes, such as the train and boiler sequences.
Baselines. We compare primarily against the method of Oh et al. [40]. neural network based approach trained on synthetic dataset. Additionally, we implement forward warp baselines that use nearest and bilinear sampling to warp pixels according to the amplified flow, along with nonparametric inpainting method [41], which we term Warp Nearest and Warp Bilinear respectively. We also compare against FLAVR [28], 3D U-Net trained for frame interpolation and finetuned on the same synthetic dataset as Oh al., albeit
constant magnification factor of 10. Finally, we provide qualitative comparisons in Section A5 of the appendix to Neural Implicit Video Representations (NIVR) [36], method that fits an implicit representation to video and displays emergent motion manipulation behavior.
Comparison with the State-of-the-Art
Visual quality. We show qualitative results on real-world videos. Figure we show still frames and closeups of results from Oh al. and our method. And in Figure we plot y-t slices through the video volume, with one dimension being time and the other being spatial?.
One qualitative finding is that while the flow network may be noisy trained with the same flow network can be much more robust. This can be seen clearly in the baby sequence of Figure 3. The forward warp methods, which depend on the optical flow estimate on the inference video, are very jittery whereas our predictions are much smoother. effect, our method distills given flow network into more robust estimator of object motion.
specific video, our model
For simplicity, refer both vertical and borizontal slices y-t slices.

---

Original
(ii) Magnifying Black Cat
(1)) No Targeting
[IV] Magntfying White Cat
(a) Original
(b) No Targeting
(c) Targeted
Figure 5: Targeted Magnification. Our network is capable targeting motion magnification. This useful when we want to focus on a specific object, as in the cal sequence above, or when we want to ignore an object that may be challenging to magnify, such as the quickly moving arm in the bookshelf sequence above.
ARFlow
ARFlow
Input (Bookshalf)
RAFT
Input (Tuning Forks)
RAFT
Input (Drum)
RAFT
Figure 6: Qualitative Comparison between Models Trained with ARFlow and RAFT. We present y-t slices from videos magnified by models trained with ARFlow and RAFT. The model (ARFlow) produces magnifications of comparable quality as the model (RAFT), despite being trained an unsupervised optical flow method.
Another point to note is that the model of Oh et al., trained using synthetic data with piece-wise linear motions, tends to suffer when the motion is complex. For example in Figure 3, with the moving fur in the cat sequence, or in Figure with the train tracks. Oh et al. also fails when the motion is extreme, as is in the case of the pole sequence in Figure where it fails to magnify an object to its full range of motion. Our model on the other hand generalizes to these methods quite well and is further improved on such out-of-domain videos with test time adaptation.
Targeted magnification. In Figure 5 we show results of targeted magnification. Given a mask, we can easily magnify just the motion that mask. This is useful when we want to focus on the motion of specific object, such in the cal sequence, or when we want to mask out an object whose motion too large to be magnified, as the case with the arm in the bookshelf sequence.
ARFlow and RAFT models. We additionally provide qualitative comparisons between method trained with the unsupervised, and less powerful, ARFlow optical flow model and our method trained with the supervised RAFT model, in Figure 6. As can be seen, despite making slightly less accurate flow estimates, the ARFlow model is fairly comparable in quality to the RAFT model. This shows that our method can enable fully self-supervised motion magnification model, in which each component is trained with unlabeled data. For all other figures in the paper, we visualize videos generated by our RAFT model.
Quantitative evaluation on real-world videos. We compare method to baselines our real- world test set in Table and Figure 7. We magnify frame pairs by magnification factors of a ranging from to 64. Because FLAVR is only trained for = 10, also evaluate this setting. We achieve the best results on almost all evaluation metrics, even with flow models that we did not train for such robustly magnify motions.
PWC-Net and GMFlow, indicating that
Quantitative evaluation on synthetic videos. Additionally, we evaluate our method and base- lines on the synthetic test sets. The subpixel subset contains frame pairs with levels of purely translational motion varying from 0.04 pixels to 1.0 pixel, and with fixed target magnified motion magnitude
pixels. The noise subset contains 21 groups frame pairs with simulated photon noise injected at increasing multiplicative factors, all with max motion of pixels and fixed magni- fied motion magnitude of 10 pixels. We report the motion error, magnification error, and SSIM in Table 2 and Figure 8. To compute the motion and magnification error, we use GMFlow [65] trained

---

24
Harp Rearest
24
Mection Error
Motinn Erre
sarp
23
Warp Blirea
worp Neares:
Dum (A-F ow)
Oh ar
550
2-3
Dh of
Dum (ART add
Dum (RAFT)
%
Sum (BAFT)
2
24
2 2-
2-
2-
2-1
Magn Ricaiion Factor a
Magnification Factor
Input Amount (px)
Amount (px)
(a) PeC-NNt
(b) RAFT (FlyingThings)
(a) Motion Error for Subpixel Test
(b) SSIM for Subpixel Test
10"
10f
Motion Ererr
Modeon Enerr
107
SSM
Hettonn
20
2
2 2
2 Magniftration Factor
21 22
10-2
10f
10
101
10-2
10-1
10
10
Kagr Regr-inctionn Factor a
Notse Factor
Noise factor
(c) GMFlow (FlyingThiings)
(d) GMFlow (Sintel)
(c) Motion Error for Noise Test
(d) SSIM for Noise Tost
Figure 7: Evaluation on real-world test set. We evaluate the methods with ranging from 1 to 64 and various flow methods. Error bars show the standard er- ror. On motion error, our method consistently obtains more accurate magnified motions.
Figure 8: Evaluation on synthetic data. We evaluate our model using the synthetic test dataset from Oh et al. [40]. We use two subsets and two metries (motion error and SSIM). Error bars show the standard error.
Table 1: Quantitative Evaluation Results on Real-world Test Set. We report the motion error of our method and baselines on our collected evaluation set. smaller motion error represents better magnification quality. For fair comparison, we show results for optical flow methods not used during training, including PWC-Net [51] and GMFlow [65] trained with FlyingThing [37]. These results are a subset of those in Figure 7. We achieve better motion magnification almost all evaluation metrics.
PWC-Net
RAFT
GMFlow
Method
acm
a=4
a=10
a=16
a=
a=2
am4
a=10
a=16
a=64
a=2
am4
a=10
ce16
cm64
Warp Nearest
0.59
1,42
4.15
7.24
35.76
0.40
0.83
2.91
5.49
28.87
0.56
1.26
3.84
6.86
33.61
Warp Bilinear
0.57
1.40
4.12
7.28
35.78
0.35
0.78
2.97
5.52
28.70
0.53
1.24
3.79
6.88
33.72
FLAVR
4.24
3.87
4.40
Oh et al.
0.51
1.37
4.29
7.60
39.49
0.47
1.21
3.86
6.95
37.53
0.52
1.40
4.34
7.64
37.31
Ours (ARFlow)
0.38
1.15
3.70
6.58
34.21
0.33
1.03
3.52
6.33
32.89
0.41
1.30
4.15
7.23
35.02
Ours (RAFT)
0.30
0.95
3.32
6.01
32.01
0.26
0.78
2.82
5.20
29.26
0.37
1.19
3.88
6.73
33.49
Table 2:
Quantitative Evaluation Results on Synthetic Videos. We report the motion error, magnification error, and SSIM of our method and baselines on two synthetic evaluation sets, Smaller motion error or magnification error, and larger SSIM indicate better quality. We use GMFlow trained with Flying Things to calculate motion error and magnification error. These results are subset of those plotted in Figure 8.
Motion Erao
J.
Magnification Eror 1
SSIM
Subpixel Text
Noise Test
Suhpicel
Subpisel Test
Noise Test
Method
0.04px
0.2px
1px
0,01x
1x
100x
0.04px
0.2px
1px
0.01x
Ix
100x
0.04px
0.2px
1px
0.01x
1x
100x
Warp Nearest
10.60
3.01
2.12
0.65
0.91
221
209 10 22.61 3.12
0.16
0.23
1.90
0.77
0.92
0.97
0.97
0.85
0.18
Warp Bilinear
10.61
3,02
2.12
0.61
0.90
226.37
209.35
22.61
3,10
0.15
0.23
1.68
0.78
0.93
0.97
0.98
0.90
0.19
Oh et al.
9.82
2.93
2.26
1.01
0.94
181.2
193.78
22.05
3.15
0.28
0.25
1.70
0.93
0.97
0.98
0.98
0.97
0.31
Ours (ARFlow)
12.29
3.26
2.35
0.82
0.70
167.77
222
16
20.87
3.05
0.20
0.18
1.69
0.75
0.89
0.94
0.97
0.94
0.10
Ours (RAFT)
10.02
2,80
2.05
164.36
2.48
0.17
0.17
0.82
0.90
0.10
with PlyingThings [37]. We achieve satisfactory results on the motion error metries and lag slightly behind on SSIM.
Discussion
We present method for learning to magnify subtle motions through self-supervised learning. We demonstrated its effectiveness through variety of experiments, using quantitative evaluations on real and synthetic data, and through qualitative results on videos containing complex motions. We also showed that the flexibility of our model allowed to be extended through test-time adaptation and targeted motion magnification. We see our work as opening new directions in visualizing subtle motions. First, while we have shown one method for defining the loss function, based on optical flow, approach could be applied to other differentiable motion estimation techniques, such emerging

---

methods for long-ram range tracking [9, 22, 21. Second, the flexibility of our model opens the possibility for other "user in the loop" extensions, beyond allowing for segmentation- magnification.
Broader impacts. Motion magnification methods have a range of applications, such as ampli- fying subtle motions in biology and engineering [57], amplifying microexpressions for assistive technology [63], assisting measurement techniques [41], and for video forensics [6. 7J. Please see our website for potential applications. It also has the potential negative use of revealing body motions that a person may think are undetectable, such as one's pulse [64, 1J. which may impinge on privacy.
Limitations. The performance of our model is closely tied to the limitations and capabilities of the underlying optical flow estimator. We share the limitation with existing work [40] that our method works by magnifying motion between the first frame and every subsequent frame, which may pose challenges in the presence of occlusion and disocclusion. The magnification loss (Eq. 3) may still be applicable in these cases, since modern optical flow models are trained to track occluded pixels. However, the photoconsistency assumption (Eq. 4) will no longer hold. For simplicity, we do not inpaint missing image regions [32], and we use distance between pixels in lieu of a perceptual loss or generative model. This simplicity has potential advantages for visualization applications, since it avoids hallucinating scene structure that is not present in the original images, but it also sometimes results in undesirable artifacts.
Acknowledgements. We thank Tae-Hyun Oh for the extensive discussions and data. We also thank Kyung-Ki Kwon, Tarun Kalluri, Long Mai, and Stella Yu for helpful discussions. Daniel Geng is supported by a National Science Foundation Graduate Research Fellowship under Grant No. 1841052. This work was supported in part by DARPA Semafor, Program No. HR001 120C0123. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.
References
[1] G. Balakrishnan, F. Durand, and J. Guttag. Detecting pulse from head motions in video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3430-3437, 2013. 10
[2] 7. Bian, A. Jabri, A. A. Efros, and A. Owens. Learning pixel trajectories with multiscale contrastive random walks. CVPR, 2022. 3, 10
[3] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High accuracy optical flow estimation based on a theory for warping. In European conference on computer vision, pages 25-36. Springer, 2004. 3
[4] M.-F Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8748 8757, 2019. 16
[5] J. G. Chen, N. Wadhwa, Y.J. Cha, F. Durand, W. T. Freeman, and O. Buyukozturk. Modal identification of simple structures with high-speed video using motion magnification. Journal of Sound and Vibration, 345:58-71, 2015. 6
[6] U. A. Ciftci and I. Demir. How do deepfakes move? motion magnification for deepfake source detection. arXiv preprint arXiv:2212. 2022. 10
[7] V. Conotter, E. Bodnari, G. Boato, and H. Farid. Physiologically- -based detection of computer generated faces video. 2014 JEEE International Conference on Image Processing (ICIP), pages 248-252. JEEE, 2014. 10
[8] A. Dave, T. Khurana, P. Tokmakov, C. Schmid, and D. Ramanan. Tao: A large-scale benchmark for tracking any object. In European Conference on Computer Vision, 2020. 6,
[9] C. Doersch, A. Gupta, L. Markeeva, A. Recasens, L. Smaira, Y. Aytar, J. Carreira, A. Zisserman, and Y. Yang. Tap-vid: A benchmark for tracking any point in a video. Advances in Neural Information Processing Systems, 2022. 10
[10] A. Dosovitskiy, P. Fischer, E. Ilg. P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 1758-7766, 2015. 1
10

---

[11] M. Elgharib, M. Hefeeda. F. Durand, and W. T. Freeman. Video magnification in presence of large motions. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 41 4119-4127. 2015. 3
[12] Everingham, S. A. Eslami, L. Van Gool, C. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 11 111:98-136, 2015. 3,
[13] H. Fan, L. Lin, Yang, Chu, G. Deng, Yu, H. Bai, Y. Xu, C. Liao, and H. Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 5374-5383, 2019. 16
[14] P. Felzenszwalb and D. Huttenlocher. Efficient belief propagation for early vision. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 1, pages I-I. IEEE, 2004. 3
[15] B. Y. Feng, H. AlZayer, M. Rubinstein, W. Freeman, and J.-B. Huang. Visualizing subtle motions with time-varying neural fields. International Conference on Computer Vision (ICCV), 2023.
[16] P. Fischer. A. Dosovitskiy, 1lg. P. Häusser, C. Hazırba, V. Golkov, P. Van der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks, arXiv preprint arXiv: 1XX.:v555400652. 2015. 3
[17] P. Flotho, C. Heiss, G. Steidl, and D. J. Strauss. Lagrangian motion magnification with double sparse optical flow decomposition. arXiv preprint 22Xiv:220..07636, 2022. 2
[18] Y. Gandelsman, Y. Sun, X. Chen, and A. Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 35:29374-29385, 2022.
[19] S. Gao, Y. Feng. L. Yang. X. Liu, Z. Zhu, D. Doermann, and B. Zhang. Magformer: Hybrid video motion magnification transformer from eulerian and lagrangian perspectives. 2022. 2,
[20] D. Geng, M. Hamilton, and A. Owens. Comparing correspondences: Video prediction with corresoondence- wise losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 3365-3376, 2022. 3
[21] A. Goyal, A. Mousavian, C. Paxton, Y.W. Chao, B. Okorn, Deng, and D. Fox. Ifor: Iterative flow minimization for robotic object rearrangement. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[22] A. W. Harley, Z. Fang, and K. Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision, 2022. 10
[23] B. Horn and B. Schunck. Determining optical flow. In Artificial Intelligence, pages 185 203- 1981. 3, 21
[24] T.-W. Hui, X. Tang, and C. C. Loy. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8981-8989, 2018.
[25] E. Ilg. N. Mayer. T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462-2470, 2017. 3
[26] A. Jabri, A. Owens, and A. Efros. Space-time correspondence as a contrastive random walk. Advances in neural information processing systems, 33: 33:19545-19560, 2020. 2,5
[27] R. Jonschkowski, A. Stone. J. T. Barron, A. Gordon, K. Konolige, and A. Angelova. What matters in unsupervised optical flow. arXiv preprint arXiv:2006.04900 2020. 1.3
[28] T. Kalluri, D. Pathak, M. Chandraker, and D. Tran. Flavr: Flow- aow-signottic video representations for fast frame interpolation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2071 1-2082, 2023. 3, 7, 15, 18
[29] A. Kirillov, Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson. Xiao, S. Whitchead, A. C. Berg. W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. (.Xiv:2304.02643 2023. 2.5
[30] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755 Springer, 2014. 6,6
11

---

[31] C. Liu et al. Beyond pixels; exploring representations and applications for motion analysis. PhD thesis, Massachusetts Institute of Technology. 2009.
[32] C. Liu, A. Torralba, W. T. Freeman, F. Durand, and E. H. Adelson. Motion magnification. ACM transactions on graphics (TOG), 24(3):519 -26,, 2005. 1, 1,2, 10
[33] L. Liu, J. Zhang, R. He, Y. Liu, Y. Wang, Y. Tai, D. Luo, C. Wang, Li, and F. Huang. Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation. In IEEE Conference on Computer Vision and Pattern Eeccognition(CVPPP 2020. 1, 2, 6, 15
[34] P. Liu, L King, M. R. Lyu, and J. Xu. Ddflow; Learning optical flow with unlabeled data distillation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 8770-877 8777, 2019. 3
[35] B. D. Lucas, T. Kanade, et al. An iterative image registration technique with an application to stereo vision. Vancouver, British Columbia, 1981. 3
[36] L. Mai and F. Liu. Motion nottionaddjustblle neural implicit video representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10738-10747, 2022. 15, 16, 18
[37] N. Mayer, E. Igg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large datasct train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4040-4048, 2016. 9, 15
[38] M. Menze and A. Geiger. Object scene flow autonomous vehicles. In Proceedings of the IEFE conference on compuder vision and pattern recognition, pages 3061-3070, 2015. 15
[39] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021. 5
[40] T.-H. Oh, R. Jaroensri, C. Kim, M. Elgharih, F. Durand, W. T. Freeman, and W. Matusik. Learning-based video motion magnification. In Proceedings of the European Conference Computer Vision (ECCV), pages 633-648, 2018 1, 2, 3,4, 4, S, 6, 7, 10, 15, 18, 19
[41] E. Pérez and E. Zappa. Video motion magnification improve the accuracy of Aision-based vibration measurements. IEEE Transactions on Instrumentation and Measurement, 71:1-12, 12, 2022. 7, 10
[42] J. Pont-Tuset, Perazzi, Caclles, Arbeláez, Sorkine- Horrinng and L. Van Gool. The 2017 davis challenge on video object segmentation. 3oXtv:/704.00677 2017. 6, 16
[43] A. Ranjan and J. Black. Optical flow estimation using spatial pyramid network. Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pages 4161-41777 2017. 3
[44] Z. Ren, J. Yan, B. Ni, B. Liu, X. Yang, and H. Zha. Unsupervised deep learning for optical flow estimation.
In Thirty-First AAAI Conference on Artificial Intelligence, 2017. 3
[45] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister Intervention-MICCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015. 2. 15
[46] G. A. Sigurdsson, G. Varol, X. Wang, Farhadi, I. Laptev, and Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 14, 2016, Proceedings, Part I 14, pages 510-526 Springer, 2016. 16
[47] E. P. Simoncelli and W. T. Freeman. The steerable pyramid: A flexible architecture for multi-scale derivative computation. In Proceedings. International Conference Image Processing, volume 3, pages 444-447. JEEÉ, 1995.
[48] Singh. Murala, and Kosuru. Lightweight network for video motion magnification. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 204 1-2500, 2023. 3
[49] D. Sun, S. Roth, and J. Black. Secrets of optical flow estimation and their principles. 2010 IEEE computer society conference on computer vision and pultern recognition, pages 2432-2439. IEEE, 2010.
3, 21
[50] D. Sun, S. Roth, and M. J. Black. A quantitative analysis of current practices in optical flow estimation and the principles behind them. International Journal of Computer Vision, 106: 137, 2014. 21
12

---

[S1] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. Pwe-net: Cans for optical flow using pyramid. warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934-8943, 2018. 3,3,9
[52] Y. Sun, X. Wang. 7. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pages 9229-9248.
PMLR, 2020. 2,5
[53] R. Szeliski. Computer vision: algorithms and applications. Springer Nature, 2022. 4
[54] Z. Teed and J. Deng. Raft: Recurrent all aal-pairs field transforms for optical flow. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part II 16, pages 402 419. Springer, 2020. 2, 6, 15
[55] Z. Teed and J. Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558 16569, 2021. 3
[56] B. Thomee, D. A. Shamma, G. Friedland, B. Flizalde, K. Ni, D. Poland, D. Borth, and I..J. Li, Vfce100m: The new data in multimedia research. Communications of the ACM, 59()):64-73, 2016. 16
[57] N. Wadhwa, G. Chen, J. B. Sellon, D. Wei, M. Rubinstein, R. Ghaffari, D. M. Freeman, O. Büyüközzürk, P. Wang. S. Sun, et al. Motion microscopy for visualizing and quantifying small motions. Proceedings of the National Academy of Sciences, 114(44): 1 (669-166444 2017. 10
[58] N. Wadhwa, Rubinstein, F. Durand, and T. Freeman. Thase-based vidco motion processing. ACM Transactions on Graphics (TOG), 32 2013. 1, 6
[59] N. Wadhwa, M. Rubinstein. F. Durand, and W. T. Freeman. Riesz pyramids for fast phase-based video magnification. In 2014 IEEE International Conference on Computational Photography (ICCP), pages 1-10, IEEE, 2014. 3,6
[60] W. Wang. M. Feiszli, H. Wang, and D. Tran. Unidentified video objects: A benchmark for dense, open- world segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10776-07855, 2021. 6, 16
[61] Y. Wang, Y. Yang, Z. Yang, I.. Zhao, P. Wang, and W. Xu. Occlusion aware unsupervised learning of optical flow. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4884 4893, 2018.
[62] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004.
[63] M. Wei, W. Zheng, Y. Zong, X. Jiang, C. Lu, and J. Liu. A novel micro-expression recognition ap- proach using bbase magnification-adapll networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2420-2424. IEEE, 2022. 3, 10
[64] H.-Y. Wa, M. Rubinstein, E. Shih, J. Guttag, F. Durand, and W. Freeman. Eulerian video magnification for revealing subtle changes the world. ACM transactions on graphics (TOG), (1)):11-8, 2012. 2, 3, 4, 6, 10
[65] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, and D. Tao. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8121-8130, 2022. 1, 3, 8, 9
[66] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision (IJCV), 127(8)1106 1125, 2019. 6. 16
[67] G. Yang and D. Ramanan. Volumetric correspondence networks for optical flow. In Advances in neural information processing systems, pages 794-805, 2019. 3
[68] L. Yang, Y. Fan, and N. Xu. The 2nd large-scale video object segmentation challenge vidco object segmentation track, Oct 2019, 6, 16
[69] F. Yu. H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE//VV conference on computer vision and pattern recognition, pages 2636-2645, 2020. 16
[70] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4471 4480, 2019. 19, 20
13

---

[71] J. Yu, A. W. Harley, and K. G. Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In Computer Vision ECCV 2016 Workshops, Part 2016.
[72] Y. Zhang, S. L. Pintea, and C. Van Gemert. Video acceleration magnification. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 529-537, 2017.
14

---

A1 Video Results
We highly encourage the reader to visit our webpage to see qualitative results and comparisons against other methods. Results on the website include:
Videos magnified at various magnification factors, and side-by-side comparisons against Oh et al. [40], FLAVR [28] and NIVR [36].
Targeted magnification results.
Test-time adaptation results.
Failure cases (see Section A8).
Ablations with regularization (see Section A6).
A2 Training Details
Architecture. We use a U-Net [45] that takes in 2 frames as input (concatenated channel-wise) and positional encoding for (described below) and outputs single frame. Our U-Net is composed of 5 layers of downsampling and 5 layers of upsampling, with skip connections between corresponding layers during downsampling and upsampling. We use a filter size of 64 at the shallowest layer, with the number of filters doubling at each downsampling, ending with a filter size of 1024 for the coarsest layer. Downsampling is performed via max pooling with a kernel size x 2 and stride 2, and the upsampling is performed using bilinear upsampling. Each layer is composed of a convolution of kernel size 3 batch normalization, and ReLU activation, all applied twice. We apply a sigmoid activation at the very end to transform the final output into the range (0, 1). Our model has a total of 17.3M trainable parameters, and the model of Oh et al. has 0.92M parameters. Note that the model of Oh et al. uses a bespoke architecture designed specifically for motion magnification, consisting of an encoder, a decoder, and a manipulator, and is fundamentally different from ours.
To encode magnification factors for input, we transform a single scalar a indicating the desired magnification factor 32 dimensional vector. We use sinusoidal positional embeddings with frequencies ranging geometrically from 2-3 to 2". In order to feed the embeddings into the U-Net we tile it spatially and concatenate it with the two input frames. The resulting input the U-Net has a channel size of 38, where 6 come from the two input frames and 32 come from the encoding. At training time the spatially tiled positional embeddings are constant-they all represent the same a value. However, at test time it is possible for the embeddings to vary spatially, allowing for various magnification factors in different areas of the video, and enabling targeted magnification.
Training parameters. The parameter Acolor is set to 10, while the magnification factors a range from 1 to 16, sampled geometrically. The learning rate is 3 x 10-4 and model training is performed using batch size of 40 with 4 A40 GPUs and an image size of 512 x 512. In terms of data augmentation, a random area is initially cropped with scale within the range of (0.7, 1.0). The cropped area is then resized to dimensions of 512 x 512. Furthermore, the image is subjected to random horizontal or vertical flipping with a probability of 0.5, well random rotation within the range of (-15s, 15%). Finally, strong color jittering is applied to the frames. These transformations applied identically to both frames. For the optical flow model, we select ARFlow [33] pretrained with KITTI 2015 [38] and RAFT [54] pretrained on Things [37]. We use 5 iterations of the RAFT model compute the flows required by our losses.
Choice of optical flow models. We implemented our algorithm using two distinct optical flow models, namely RAFT [54] and ARFlow [33]. During the training process, we kept the pretrained optical flow model weights fixed, employing them solely to provide motion estimates for our loss computations and gradients for backpropagation unsupervised model, to demonstrate that our method works in a truly self-supervised manner, in which no component is trained with labeled data. We then use RAFT, strong supervised flow network, to train a high quality, weakly-supervised magnification model that serves as a rough upper bound to the performance of a fully self-supervised model. We note that the comparison of quantitative results of both models indicates that our method has better performance with better optical flow quality, hinting that our framework can be benefit from future improvement of optical flow models.
train the U-Net. We use ARFlow, an
15

---

Table A1: Dataset Statistics (Train Set). We report basic statistics of training dataset, including the temporal sampling stride and the number of frame pairs before and after filtering.
YouTTue- vOS DAVIS Vimeo-90k
TAO
UVO
Total
Sampling Stride 1
5
# Raw Pairs 46,858 4,050
3,089
193,836
176,092
175,434
71,836
737,975 9,749 145,601
70,830
# Seleted Pairs
370
28,734
50,441
29,795
22,462
A3 Dataset Details
Dataset overview. To train and evaluate our model, we compile large dataset of frame pairs by sampling from various existing datasets: Youtube- -ouuubb--VO-201999 [68], DAVIS [42], Vimeo-90k [66]. Tracking Any Object (TAO) [8], and Unidentified Video Objects (UVO) The train set is collected by sampling from the train sets of the above datasets, with the exception of DAVIS, where we use the trainval split due to its smaller size. To construct real-world test set, total of 50 frame pairs are randomly collected from each dataset's ss test set, except for the TAO dataset, in which the validation set is used. Because the TAO dataset is comprised of five distinct datasets (ArgoVerss [4], BDD-100K [69], Charades [46], LaSOT [13], and YFCC100M [56]), we sample 50 frame pairs from each of these sub-datasets. In total, 650 frame pairs are acquired across all datasets for evaluation purposes, Basic statistics of the collected dataset is listed in Table Al.
Sampling and filtering. We use consistent filtering and sampling approaches for both the training dataset and the real-world test set. We use two sampling methods, characterized by different temporal strides (1 or 5) indicating the temporal distance between two sampled frames. Specifically, stride of 5 means that frame i + 5 is sampled to form a frame pair alongside frame i. When using stride of 1, non-overlapping consecutive frame pairs are sampled. For the datasets Youtube- Noutube-VOS-200 V0S-2019, DAVIS, and Vimeo-90k, stride of is used. In the case of TAA and UVO, strides of 1 and 5 are utilized to supplement the dataset with additional samples due to the high framerates these datasets.
After obtaining raw frame pairs from these datasets, a filtering process is conducted to eliminate frame pairs exhibiting significant object or camera motion. We estimate motion using RAFT and set upper bounds for different quantiles of the per-pixel flow magnitude distribution. These include an upper bound of 20 on the 99.9th percentile, an upper bound of 2 on the 80th percentile, and an upper bound of 0.1 on the 0.0 percentile. We find empirically that these thresholds result in frame pairs that have little camera motion and small object motions. To filter out identical or near-identical frame pairs, an additional lower bound is introduced to filter out frame pairs with mean squared error (MSE) of less than 10 between the two frames.
After sampling and filtering, we obtain a training dataset of 145k frame pairs. Detailed information regarding the number of sampled raw pairs and the number of selected pairs can be found in Table A1 in our paper.
A4 Figure Parameters
We give comprehensive summary of the magnification factors used and location of y-t slices or closeups for each sequence in Table A2 and Table A3. Additionally, we also provide the corresponding figure index in our paper, and the frame index, indicating which frame we displayed in our figures. Note that some images are cropped or stretched to optimize their presentation within the figures.
A5 Additional Qualitative Results
Comparison with NIVR. Neural implicit video representations (NIVR) [36] is a method that encodes video into neural implicit representation. Additionally, the method uses a phase- based positional encoding which allows for control over motion. However, the method does not give precise control over magnification factors as the manipulation is over the latent positional encodings, and is only shown to perform motion magnification up to factor of about (our work and previous works can achieve factors of up to 200). Furthermore, the code for their method is unreleased. Therefore we compare our method against their results obtained through correspondence with the authors. We show y-t slices in Figure AI. We achieve comparable performance y-t slices for the flower and guitar
16

---

Table A2: Information on y-1 Slices of Magnified Sequences. We represent the location of y-1 slices with the upper left (11, 91) coordinate and bottom right coordinate (22, y2) for every sequence appeared in our paper. We also provide magnification factors for each y-t slices. For frames with two y-t slices in the same frame, we report the left and right locations accordingly, and mark with "L" for left, "R" for right in the table. For sequenees magnified by different magnification factors, we report information for every motion magnification factor.
Sequence
Drum
Pole
Trming Yorks (L)
Tuning Forks (R) Baby
Cats (L)
Figure
Figure 1,6
Figure 3,3
Figure 1, 6, A3
Figure I, 6, A3
Figure 3
Figure 5
5
20
5
20
20
Upper Left
(180. 250) (20. 120)
(175, 200)
(340, 260)
(510, 212) (280. 160)
Bottom Right
(181, 310)
(140, 121)
(315, 201)
(480, 261)
(511, 292) (281, 300)
Sequence
Cats (R)
Bookshelf (L.)
Bookshelf (R)
Flower
Gaitar!
Camel
Figure
Figure 3,5
Figure 5,6
Figure
Figure A1
Figure Al Figure Al 2
20
15
15
2
Upper Left
(710, 130) (240, 340)
(560. 260)
(260) 50)
(20, 120) (250, 90)
Bottom Right
(711, 270) (241, 400)
(561, 320)
(350, 51)
(21. 160)
(280, 91)
Soquence
Cats (R)
Bookshelf (L)
Boiler
Flower
Guitar2
Figure
Figure A2
Figure A2
Figure A3.A4
Figure A5
Figure A5 25
10
10
30
20
Upper Left
(710, 130) (240. 340)
(100, 200)
(240, 10)
(280, 50)
Bottom Right (711, 270)
(241, 400)
(200, 201)
(241, 50)
(281, 100)
Table A3: Information on Closeups of Magnified Sequences. We represent the location of closeups with the upper left (æ1, y1) coordinate and bottom right coordinate (2). We include frame index that appears in our figures, and frame index starts from for video. We also provide magnification factors for each closeups in our paper.
Sequence
Train
Pole
Timie
Camel
Boiler
Figure
Figure
Figure
Figure 4
Figure
Figure 4
Q:
20
20
20
20
30
Upper Left
(20, 520)
(0, 0)
(50, 520)
(280, 50)
(100, 120) (36),
Bottom Right
(300. 710)
(200, 200)
(240. 710)
(420. 300)
Frame Index
417
1. 190
417
30
19
Sequence
Train
Timing Forks Boiler
Boiler
Camera
Figure
Figure A2
Figure A3
Figure A3
Figure A4 Figure A5 30
30
Upper Left
(50. 520)
(80, 120)
(0. 0)
Bottom Right (240. 710)
(280, 320)
(45, 200)
Frame Index 417
10
19
16
Table A4: Quantitative Evaluation Results on Real-world Videos. We report full evaluation results on real-world videos. Performance is measured by motion error and magnification error for various optical flow estimators.
Motion Error
Magnification Error
coml
cc22
am4
a=8
a= 16
cr232
a=64
a=1
a=2
o=4
a=8
cm16
a=32
a=64
Warp Ncarest
0.32 0.59
1.42
3.16
7.24
16.24
35.76
0.56
1.07
2.28
4.65
9.54
19.94
43.51
Warp Bilincar
0.28
0.57
1.40
3.16
7.28
16.18
35.78
0.63
1.08
2.21
4.51
9.45
19.82
43.37
PWC-Net
Oh et al.
0.05 0.51
1.37
3.27
7.60
17.52
39.49
0.20
0.80
2.06
4,51
9.65
21.58
50.36
Ours (ARFlow)
0.04
0.38
1.15
2.81
6.58
15.18
34.21
0.13
0.74
2.13
4.52
9.19
19.68
41.51
Ours (RAFT)
0.05 0.30
0.95
2.48
6.01
14.05
32.01
0.13
0.55
1.56
3.62
7.74
16.22
34.82
Warp Nearest
0.24 0.40
0.83
2.19
5.49
12.71
28.87
0.65
1.18
2.25
4.33
8.79
18.04
39.63
Warp Bilincar
0.18 0.35
0.78
2.20
5.52
12.81
28.70
0.76
1.15
2.08
4.46
9.01
17.90
38.22
RAFT
Ohet al.
0.05 0.48
1.21
2.92
6.94
16.39
37.55
0.29
0.83
1,90
4,20
21.44
51.07
Ours (ARFlow)
0.04 0.33
1.03
2.64
6.33
14.63
32.89
0.19
0.75
2.06
4.68
10.25
22.51
47.28
Ours (RAFT)
0.04 0.26
0.78
2.10
5.20
12.66
29.26
0.16
0.56
1.58
3.79
8.34
17.49
35.59
Warp Nearest
0.27 0.56
1.26
2.93
6.86
15.43
33.61
0.61
1.20
2.52
5.14
10.43
21.17
44.56
GMFlow
Warp Bilinear
0.24 0.53
1.24
2.94
6.88
15.46
33.72
0.64
1.17
2.42
5.05
10.38
21.44
45.14
(FlyingThings))
Oh et al.
0.09 0.52
1.40
3.31
7.64
16.84
37.31
0.47
0.92
2.11
4.75
10.53
22.74
51.99
Ours (ARFlow)
0.07 0.41
1.30
3.18
7.23
16.68
35.02
0.28
0.82
2.18
4.64
9.72
21.47
44.74
Ours (RAFT)
0.07 0.37
1.19
2.98
6.73
15.32
33.49
0.26
0.65
1.81
4.36
9.11
19.62
42.17
Warp Nearest
0.26 0.55
1.43
3.64
8.02
17.32
36.69
0.66
1.36
3.25
7.73
16.76
33.61
66.49
GMFlow
Warp Bilincar
0.23 0.51
1.39
3.45
8.04
17.19
36.84
0.96
1.70
3.57
7.79
16.64
33.53
66.57
(Sintel)
Oh et al.
0.06 0.53
1.49
3.60
8.13
17.53
37.31
0.65
1.09
2.40
5.26
11.28
24.01
51.71
Ours (ARFlow)
0.05 0.39
1.38
3.49
7.87
17.25
36.10
0.26
0.96
2.91
6.45
13.27
28.34
55.73
Ours (RAFT)
0.06 0.39
1.24
3.22
7.43
16.33
34.44
0.28
0.72
2.03
4.94
10.88
23.45
47.67
17

---

Original
Original
NIVR
Original
NIVR
Ours
Input (Flower)
Ours
Input (Guitar1)
Input (Camel)
Figure AI: Qualitative Comparison with NIVR. NIVR [36] magnities learned implicit representation, which does not map cleanly to precise magnification factor. We found using a with our method gave comparable magnification to the results from NIVR. Because there is no publicly released code for NIVR. we compare against videos and results from the NIVR website. Above we visualize the y-f slices of the original videos, NIVR videos, and our videos for comparison.
FLAVR
FLAVR
FLAVR
Input (Cats)
Ours (TTA)
Input (Bookshelf)
Ours
Input (Train)
Ours
Figure A2: Qualitative Comparison with FLAVR. We use a 10 for our method and compare the results
with the magnified videos of FLAVR. We show the y-t slices of FLAVR videos and our videos. We also show closeup in the train sequence, shown in the green rectangle.
Table A5: Quantitative Evaluation Results on Synthetic Videos. We report the evaluation results the sub-pixel test with synthetic videos with all different imput motion amount, with three metrics including motion enror, magnification error, and SSIM.
Motion Firor
Input Motion
0.04px
0.05px
0.06px
0.08px
0.10ps
0.1
0. 0.16px
0,20px
0.25ps
0.32px
0.40pp
0.50pp
0.63px
0.79px
1.00px
Warp Neareat
10.60
6.80
5.45
4.77
4.13
3.80
3.01
2.80
2.56
2.57
2.49
2.21
2.24
2.12
Warp Bilinear
10.61
8.34
6.30
5.44
4,77
4,13
3.58
3.02
2.80
2.56
2.57
2.49
2.20
2.23
2.12
Oh
9.82
2.93
Ours (ARFlow)
12.29
9.86
8.16
6,38
5.37
4.48
3.99
3.26
3.02
2.65
2.69
2.53
2.42
2.42
2.35
(RAFT)
10.02
2.80
2.05
Magnification Error
Input Motion
0.04px
0.05px
0.06px
0.08px
0.10px
0.13px
0.16px
0.20px
0.25px
0.32px
0.40px
0.50px
0.63px
0.79px
1.00pxx
Warp Nearest
209.10
155.27
117.06
85.36
62.23
45.11
32.52
22.61
16.61
12.75
10.71
7.98
5.29
4.59
3.12
Warp
209.35
116.98
45.37
12.73
7.99
Oh et al.
193.78
145.86
106.22
79.79
56.42
42.23
30.96
22.05
17.37
12.50
10.53
7.90
5.55
4.39
3.15
(ARFlow)
222.16
165.20
42.24
Ours (RAFT)
175.45
133.33
97.06
69.86
52.10
36.56
25.42
17.01
13.36
9.86
8.23
5.94
4.20
3.32
2,48
1
Input Motion
0.04px
0.05px
0.06px
0.08px
0.10px
0.13px
0. 16p
0,20px
0.25px
0.32px
0.40pp
0.50pp
0.63px
0.79px
1.00px
Warp Nearest
0.77
0.80
0.82
0.86
0.89
0.90
0.91
0.92
0.93
0.94
0.95
0.95
0.96
0.96
0.97
Warp Bilincar
0.78
0.81
0.83
0.86
0.89
0.91
0.91
0.93
0.94
0.94
0.95
0.95
0.96
0.97
0.97
Oh al.
0.93
0.95
0.97
0.97
0.98
0.98
Ours (ARFlow)
0.75
0.77
0.79
0.82
0.84
0.86
0.87
0.89
0.89
0.90
0.92
0.92
0.93
0.93
0.94
Ours (RAFT)
0.82
0.84
0.86
0.89
0.90
0.90
0.94
sequences, and our y-t slice for the camel sequence is crisper than NIVR. Comparison of videos is available on our webpage.
Comparison with FLAVR. We also compare against FLAVR [28], 3-D U-Net proposed for
frame interpolation and downstream tasks including motion magnification. Following correspondence with the authors, we replicate their experiments and finetune the public frame interpolation FLAVR checkpoint for motion magnification by training subset of the synthetic dataset of Oh al. [40] with a se 10. The results of this comparison are displayed in Figure A2. Our magnification has crisper boundary and clearer visual quality. Comparison of videos available on our webpage.
A6 Additional Experiments
Full evaluation results. In our main paper, due to space limitations we presented selected groups of evaluation results in tables. We present all evaluation results in the following tables for reference,
18

---

w/o regularization
w/ regularization
wo regularization
w/ regularization
Figure A3: Qualitative Comparison with No Regularization Experiments. We show magnified frames and g-t slices from our model with and without the color loss regularization.
Table A6: Quantitative Evaluation Results Synthetic Videos. We report the evaluation results on the noise test with synthetic videos with all different noise factors, three metrics including motion error, magnification error, and SSIM.
Motion Ervor
Noise Factor
0.01
0.02
0.03
0.04
0.06
0.10
0.16
0.25
0.40
0.63
1.00
1.59
2.51
3.98
6:31
10001
15.85
25.12
39.81
63,10
100,00
Warp
0.65
0.76
0.76
Warp Bilincar
0.61 0.62
0.66
0.62
0.61
0.63
0.68
0.73
0.73
0.74
0.90
1.22
2.21
1.84
2.51
12.85
31.72
78.00
153.10
182.60
226.37
Oh er
1.01 1.00
1.05
0.94
0.97
0.96
0.96
0.96
0.96
0.90
0.94
1.07
1.14
1.45
2.18
8.92
17.20
$1.44
119.34
163.99
181.26
Ours (ARFlow)
0.74
6.44
Ours (RAFT)
0.69
0.71
0.66
0.68
6.32
44.04
107.10
Magnification Error
Noise Factor
0.01
0.02
0.03
0.06
0.10
0.16
0.25
0.40
0.63 1.00
1.59
2.51
3,98
10.00
15.85
25.12
39.81
63.10
100.00
Warp Nearest
0.16 0.17
0.16
0.16
0.15
0.16
0.18
0.17
0.20
0.20
0.23
0.42
0.93
1.13
0.84
2.22
6.51
11.34
9.65
2.74
1.90
Warp Bilineur
0.15 0.17
0.16
0.15
0.14
0.16
0.17
0.17
0.19
0.19
0.23
0.39
0.58
0.51
0.74
2.21
6.19
9.40
8.47
2.68
1.68
Oh
0.28
0.30
0.25
0.26
0.34
0.25
0.30
1.10
1.72
4.00
3.45
(ARFlow)
0.19
0.19
0.08
1.48
1.97
Ours (KAFT)
0.17
0.17
0.18
0.17
0.17
0.17
0.17
0.17
0.17
0.16
0.17
0.21
0.22
0.24
0.31
0.8
1.32
2.28
1.70
1.50
1.49
SSIM 1
Noise Factor
0.01 8.02
0.10
0.16
0.25
0.40
0.63
1.00
1.59
2.51
3.98
10000
15:85
25.12
39.81
63.10
100.00
Warp Nearest
0.97 0.97 0.97
0.97
0.97
0.97
0.97
0.96
0.95
0.92
0.85
0.75
0.59
0.42
0.29
0.20
0.14
0.14
0.15
0.18
0.18
Warp Bilinear
0.98 0.98
0.97
0.97
0.96
0.94
0.90
0.83
0.30
0.27
0.19
0.17
Oh
0.98 0.98
0.98
0.98
0.98
0.98
0.98
0.98
0.98
0.98
0.97
0.96
0.93
0.66
0.55
0.45
Ours (ARHiow)
0.97
0.97 0.96
0396
0.97
0.97
0.97
0.97
0.96
0.96
0.94
0.92
0.86
0.74
0.61
0.46
0.32
0.21
0.15
0.13
0.10
Ours (RAFT)
0.97 0.97
0.97
0.97
0.97
0.97
0.96
0.96
0.94
0.91
0.84
0.73
0:43
0.30
0,20
0.14
0.12
0.10
including results on real-world videos with different optical flow methods in Table A4 and on all subsets of the synthetic data from Oh el al. [40] in Table A5 and Table A6.
No regularization. To demonstrate the significance of the color loss, Loolors we ablate it out and show qualitative results for comparison. Figure A3, we compare our method with and without a color loss with frames from the magnified videos. Notably, we observe that although the magnified motion appears similar to our method with regularization, the color of the results obtained with Azolor 0 significantly deviates from the correct colors and has many artifacts.
A7 Forward Warp with Inpainting
Implementation. We introduced two forward warp techniques, which we refer as Warp Nearest and Warp Bilinear, as the Lagrangian magnification baselines. To implement these methods, we initially computed optical flow using the same model applied in our experiments (RAFT trained with FlyingThings)) Subsequently, we employed forward warping in either nearest or bilinear manner, applying optical flow values scaled by magnification factors to relocate pixels within the image. Furthermore, utilized nonparametrie inpainting with the built-in functions from OpenCV to address areas left blank due to pixel relocation.
In addition to using nonparametric inpainting, we explored the option of inpainting with DeepFillv2, a powerful learning- based inpainting method proposed by Yu et al. [70]. Two different inpainting methods imply that inpainting is not the bottleneck for Lagrangian magnification without deep learning, thus highlighting the advantage of our method. Our evaluation results indicate comparable performance between forward warp techniques with different inpainting methods. Since the forward warp method employing nonparametrie inpainting yielded slightly better qualitative and quantitative results, used forward warping with nonparametric inpainting in our main paper.
19

---

(i) Original Framn
(ii Warp Nearest
(ili) Warp Bilinear wo Inpainting
(iv) Warp Nearest w Inpainting
(v) Warp Bilinear w Inpainting
(vi) Ours
wo Inpainting
Figure A4: Failure Cases of Forward Warp Baselines. We show one frame from the boiler sequence, along with the magnified frame from warp nearest/bilincar without inpainting, warp ncarest/bilincar with nonparametric inpainting. The closeups and y-t slices obtained through forward warp methods with and without inpainting exhibit difficulties in managing object boundaries and the background. contrast, our method successfully produces a satisfactory magnified frame under these challenging conditions.
Table A7: Evaluation on Real-world Videos for Different Inpainting Methods for Forward Warp. Similar to Table I, we evaluated and compared among different inpainting methods for both warp nearest and warp bilinear, including no inpainting (labeled as "None"), nonparametric inpainting, and DeepFillv2 [70] inpainting.
PWC-Net
RAFT
GMFlow
Method
Inpainting
o=2
a=4
a=10
a=16
a=64
a=2
a=4
a=10
a=16
a=64
a=2
a=4
a=
10
a=16
a=64
None
0.92
1866
5.43
9.12
42.23
0.50
1.18
3,87
7.13
39.82
0.91
1.70
5.29
9.05
50.38
Warp Nearest
Nonparametric
0.59
1.42
4.15
7.24
35.76
0.40
0.83
2.91
5.49
28.87
0.56
1.26
3,84
6.86
33.61
DeepFillv2
0.61
1.40
4.21
7.44
36.20
0.39
0.84
2.96
5.47
29.21
0.55
33.63
None
0.68
1.49
4.32
7.44
36.93
0.39
0.91
3.07
5.79
30.60
0.66
1.62
4.26
7,40
35.71
Warp Bilinear Nonparametric
0.57
1.40
4.12
7.28
35.78
0.35
0.78
2.97
5.52
28.70
0.53
1.24
3.79
6,88
33.72
DeepFillv2
0.57
143
4.14
7.34
35.93
0.34
0.78
2.96
5.50
28.95
0.53
1.24
3.92
7.04
33.45
Table A8: Evaluation on Synthetic Videos for Different Inpainting Methods for Forward Warp. Similar to
Table 2, we evaluated and compared among different inpainting methods for both warp nearest and warp bilinear, including no inpainting (labeled as "None" in this Table), nonparametric inpainting, and DeepFillv2 [70].
Motion Broor
Magnification Error
Subpinul Test
Test
Subgisel Test
Noise
Subpixel Test
Noise Test
Method
Impainting
0.04px
0.2px
1px
0.01x
1.0x
100.0%
0.04px
0.2pt
1px
0.01x
1.0x
100.0x
0.0-ppx
0.2px
1px
0.01x
1.0x
100.0x
None
36.20
3.74
2.25
0.71
0.89
207.76
1238 97
28.68
3.19
0.16
01:33
1.95
0.19
0.44
0.75
0.87
0.71
0.03
Warn Nearest
Nonparametric
10.60
3001
2,12
0.65
0.91
221.32
209 10
22.61
3,12
0.16
11.23
1.90
0.77
0.92
0.97
1199
0,85
0.18
DeepFillv2
10.66
3,00
2.13
0.66
0.91
220.66
206.04
22.39
3.13
0.16
0.21
2.06
0.72
0.91
0.97
0.97
0.85
0.14
None
0.64
12.56
1,88
0.94
0.94
0.86
0.04
Warp Bilinear DeepFillv2
Nonparametrcc
236.37
22.61
3,10
1.68
0.98
0.90
10.56
3,02
2.12
0.62
0.90
225.30
207.24
22.64
3.11
0.15
0.23
2.30
0.77
0.93
0.97
0.38
0.90
0.09
Table A7 and Table A8 present a detailed comparison between the forward warp baselines without inpainting and with the two inpainting methods. In our evaluation of real-worhd videos, observed notable decrease in motion error when using inpainting methods in comparison to the warp baselines without inpainting,
expected. On the other hand, during noise tests, especially when using large noise factors, the warp baselines without inpainting exhibited better performance. Notably, we detected increase in SSIM with noise factor increasing (see Table A6 for more details), for warp baselines with inpainting in noise tests. Our examination revealed that warp baselines produced distorted frames with significant noise factors and struggled to magnify motion when input motion was minimal. Consequently, the evaluation for forward warping baselines may not be able to reflect the actual performance in such cases. Therefore, given the slight advantage of nonparametric inpainting on real-world evaluations, we show the baselines with nonparametric inpainting in visualizations.
Discussion. The two forward warp methods, implemented with advanced techniques as the motion estimator and the inpainter, serve as powerful baselines for Lagrangian techniques, effectively estab- lishing an upper performance limit for straightforward Lagrangian methods. The magnified frames produced through forward warping closely resemble the input frames, thus exhibiting favorable image quality due to their inherent implementation characteristics. Nevertheless, our method consis- tently outperforms or matches these baselines across various evaluation metrics. Upon qualitative assessment, approach excels well, particularly in complex scenarios involving occlusion and
20

---

Original
Original
Oh at
Input (Flowar)
Ours
Input (Guitar2)
Ours
Input (Camera)
Ours
Failure Cases. We report several typical failure of thin structures background artifacts. See Section for details.
Figure A5:
disocclusion or challenging flow patterns. We provide one challenging case in Figure A4, where large motion and occlusion were involved. While the results from forward warping methods with/without inpainting suffer from handling the object boundary and background, our method still capable to provide satisfactory magnified frame.
A8 Failure Cases
While our method demonstrates favorable performance in most videos, there are cases where it falls short. One particular failure mode occurs when video contains thin structures such tree branches and guitar strings. In these cases it appears the flow network has incorrectly estimated the optical flow during training, causing our model to either "bleed" motion into the stationary background or fail to magnify the motion at all (see flower and guitar in Figure A5). The failure may be attributed to smoothness assumptions of flow estimators [50, 49, 23], and we expect results improve optical flow methods get more accurate. Another common failure is incorrectly magnifying background that has zero flow. This happens when small errors from the optical flow model magnified, as shown in the camera sequence in Figure A5.
21


Reviews:
[{"review_id": "X6bsexT7Ul", "reviewer": "Reviewer_XnFN", "summary": "This paper focuses on learning a pair-wise motion magnification model in a self-supervised manner. The authors employ recent optical flow models to estimate the flow fields between the original and the motion magnified image pairs. The UNet concatenates a sinusoidal encoded magnification factor with the original images and generates the magnified image.\n\nThe learning process of the UNet is facilitated by a loss function that enforces consistency between the original and magnified flow fields, as well as the consistency between the backward warped images. To demonstrate the effectiveness of the proposed method, the authors curate a large-scale real-world training set. They conduct evaluations both quantitatively on a synthetic dataset and qualitatively on real-world data. The results demonstrate superior performance over previous supervised methods learned on synthetic data.", "questions": "- In lines 83-84, the authors claim that the method's capabilities are more similar to those of Lagrangian methods. However, the internal workings of the model remain unknown. It would be interesting if the authors could conduct some analysis, such as investigating whether some UNet layers implicitly contain motion cues.\n\n- The warping loss compares warped images. What would happen if we additionally, or only, compare two warped images with I_0?\n\n- In the supplementary video, is the magnification factor applied only to the ground in the jumping sequence?", "limitations": "The limitations have been addressed adequately in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well organized and written. Sec.1 introduces the problem effectively and motivates the design of the method. Sec.2 provides a brief but comprehensive review of the previous methods. In addition to the overall organization, sufficient details (including the code) are given for a better understanding of the method, such as the footnote on page 5.\n\n- The method itself is simple and effective:\n\n  - A simple UNet is a compact solution that avoids complicated operations, e.g., explicit optical flow estimation and inpainting.\n\n  - The magnification factor is concatenated with the input image pair after sinusoidal encoding, which enables regionally varying magnification. This is difficult for a single magnification factor as in [33].\n\n  - The self-supervised learning losses enable online adaptation to a specific sequence for better quality.\n\n- The evaluation is comprehensive and achieves significant improvement over previous methods.\n\n- The curated large-scale real-world dataset will encourage further research in this topic.", "weaknesses": "I do not see significant weakness of the paper since it is simple and effective. The only missing piece I come up with is that since the model itself is simple, the author could make some deeper analysis of a learned UNet to understand the underlying mechanism of the model.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your thoughtful and positive review. We appreciate that you believe our method to be simple yet effective, and are glad that you found our paper well-organized and well-written. Below, we address some of your questions and suggestions.\n\n**Analysis of UNet**\n\nWe agree with you that a deeper analysis of the UNet could bring about better insight to how the model encodes motion information. Given that our model is able to successfully magnify motion there must be an understanding of this in the network. However, analyzing and interpreting internal activations of neural networks is still an open research problem. We believe our paper should focus primarily on our motion magnification objective, but are hopeful that our work could be a useful case study for future feature visualization or mechanistic interpretability research.\n\n**Lagrangian Label**\n\nWe term our method Lagrangian because our loss explicitly tracks pixels through a video and tries to magnify those tracks, and not because of how the neural network behaves. This is as opposed to Eulerian approaches which aim to magnify motion by representing and manipulating motion as spatially fixed variables over time. These terms derive from Lagrangian and Eulerian frames of reference in the field of fluids.\n\n**Reformulation of the Loss**\n\nGiven a generated image, which we hope to be motion magnified, our color loss warps this generated image such that it matches the target reference image $I_0$. We then take a photometric loss between the warped generated image and the reference image, with the goal of encouraging the network to generate images that have similar color to the reference image. To be consistent with prior work like DeepMag, our network only predicts a single magnified frame, but it should be possible to predict multiple frames. In this case we would compare two or more warped frames to the reference image. We do agree though that there is much room for extensions to our loss function for future work. Please let us know if this answered your question or if you would like anything else clarified.\n\n**Supplementary Video**\n\nThe \"jumping\" clip appears twice in the supplementary video. In the first instance the magnification is applied to the entire clip, not just the ground. In the second instance we target the model to magnify the ground but not the legs. We remove magnification of the legs in the \"jumping\" clip because the motion of the legs is too large to be magnified reasonably at a magnification factor strong enough to see the ground shake.\n\n&nbsp;\n\nAgain, thank you so much for your positive review, as well as your questions and feedback. We look forward to answering any more questions you may have."}, {"review_id": "2mJn2DbcPI", "reviewer": "Reviewer_nhiY", "summary": "This paper uses the classical method of Lagrangian to self-supervise the task of motion magnification. Thanks to the proposed self-supervision-based technique, the proposed method can also be adapted during the test time. As shown in Figure.1, the proposed method is simple, where the optical flow vectors of videos before and after magnification are compared.  The optical flow of the motion amplified video is compared to the scaled (by the amplification factor) optical flow of the original video, to derive the magnification loss. To make the output video color consistent, color loss is also used. Videos are provided in the supplementary material for qualitative analysis.", "questions": "How does the proposed method behave with low-quality optical flow? Does the method improve its performance in difficult videos, when trained on a larger amount of data? I can imagine that when trained on a large collection of videos, the method may generalize and hence also perform well in some difficult cases. ", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe method presented in this paper is simple, straightforward, and meaningful. \n2.\tThe experimental evaluations validate the proposed method. Supplementary videos are helpful.\n3.\tThe source code is also provided in the supplementary, which further highlights the simplicity.\n4.\tLimitations of the method are well discussed, and failure cases are shown.\n5.\tThe paper is well-written and easy to follow.\n", "weaknesses": "1.\tThe proposed method largely depends on the pre-trained optical flow network. \n2.\tGiven the nature of the addressed problem, its evaluation is known to be difficult. This is reflected in the experiments. \n3.\tThe experiments are conducted in a relatively small amount of video frames, and the paper discusses “out of the distribution” and “test time adaptation”. It would be interesting to see how the method generalizes when training on a large number of videos, before proceeding to discuss the rest.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and well thought-out comments. We would like to address the questions that you have:\n\n**Pre-Trained Optical Flow**\n\nWe agree with the assessment that our model’s performance is closely tied to the performance of the underlying flow model, and will add a discussion of this in our limitations section. But we also note that because our method is agnostic to the form of tracking used (as long as it is differentiable), it is also able to take advantage of future improvements in optical flow models.\n\n**Low Quality Optical Flow**\n\nOur goal was to produce the best self-supervised motion magnification method possible. As such we use one of the best off-the-shelf optical flow algorithms, RAFT, and did not conduct any experiments analyzing the consequences of using worse flow. We would guess that with lower quality flow the motion magnification capabilities of models trained with our loss will worsen. With poor enough flow, training may not even converge. Conversely, with better flow we expect our method to correspondingly get better.\n\n**Larger Amount of Data**\n\nWe agree with you and believe that more data should help the performance and generalization of models trained with our loss. For example, in our paper we train on a decently sized dataset of frame pairs. However, the distribution of this data is not broad enough to cover all of our test videos and we therefore see that test time adaptation can further improve the quality of our magnification results. With diverse enough data we believe that test time adaptation would become less necessary, or provide a smaller benefit, as our model generalizes better.\n\n&nbsp;\n\nThank you for your thoughtful questions, and please let us know if you have further questions or need further clarification."}, {"review_id": "RB0VWOBgy1", "reviewer": "Reviewer_uXCg", "summary": "The paper introduces an optical-flow-based lagrangian motion magnification method, learned through self-supervised learning. The architecture is very simple -- just a U-Net that inputs two temporally consecutive frames and outputs a motion-magnified image. To train the U-Net, the method uses an off-the-shelf optical flow method, estimates the motion between two frames, and considers the motion as real motion of the scene. Then it penalizes the difference between the estimated magnified motion (constant * estimated motion) and the motion between the reference image and motion-magnified image (i.e. output image). The method demonstrates both good quantitative and qualitative results.", "questions": "- Usage of the discriminator loss? I wonder if using an adversarial loss from GAN would help more realistic appearance where flow is not accurate, occlusion happens, or artifacts occur.\n- I wonder if the collected data or at least the information on train/test splits will be available if the paper is accepted.", "limitations": "- Probably another limitation would be that the method's success depends on the off-the-shelf optical flow and segmentation methods.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Comprehensive related work\n\n  The paper provides a comprehensive literature survey, which helps understand previous related work and where this paper positions among them.\n\n+ Implementation details\n\n  Sec. 4.1, Sec. 4.2, and supplementary material provide sufficient implementation details so that it's easy to understand the choices of hyper-parameters, training configuration, dataset curations, and training details. Parts of source codes are also included in the supplementary material, which all help reproduce the proposed methods. \n\n+ In-depth evaluation\n\n  Fig. 5, Fig. 6, Table 2, and Table 3 provide in-depth evaluation of the proposed method and related methods on real-world and synthetic videos. Given that there is no public benchmark and the difficulty of evaluation on this topic, the paper tries its best on providing sufficient evaluation.", "weaknesses": "- How to handle occlusion and disocclusion?\n\n  It seems already stated in the limitation section, but I wonder if the method doesn't explicitly handle occlusion and disocclusion. If it doesn't, then can it be a problem? Does U-Net learn to handle them to some extent? When watching the supplementary video, the model doesn't seem to output hallucinated appearance around the disoccluded region, which seems good.\n\n- Worse SSIM in Table 3\n\n  In Table 3, compared to DeepMag, all metrics are better except for SSIM. I wonder why it's the case. What makes the DeepMag's SSIM better than the proposed method?\n\n- Moving background?\n\n  In the supplementary video (1m14s and 2m12s), I am wondering why the background has motion and it's moving? Is it due to that the optical flow method hallucinates motion in the background and it's used during the training? Can this problem be resolved without using the target segmentation mask?\n  By the way, this is another question: what if $L_{mag}$ in Eq. (3) is applied to the target segmentation objects only and penalizes the background motion to be zero? Can it produce better results and prevent the background from moving?\n\n\nThere are some unresolved concerns but the strength outweighs the weaknesses for now. I would like to give Borderline Accept for now, but the rating could change after the discussion phase.\n\n---\nAll concerns are resolved. Thus I am updating my rating to 7. Accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive and comprehensive assessment of our paper. Below, we address your questions and concerns:\n\n**Occlusions and Disocclusions**\n\nThis is an excellent observation. As noted our model produces qualitatively good results even in the presence of occlusions and disocclusions. This is because optical flow models are already designed to handle occlusions and disocclusions well, so our magnification loss also incentivizes correct predictions in these scenarios. \n\nThe other component in our loss, the color loss, can be affected by occlusions and disocclusions. To mitigate this, in early experiments we tried masking occlusions with our loss, where we check for occlusions by computing flow both forwards and backwards and performing a cycle consistency check. We apply this occlusion mask to the color loss such that pixels that are occluded or disoccluded do not contribute to the loss. We found that results did not change significantly with occlusion masking, possibly because the magnification loss already handled occlusions and disocclusions adequately, and we therefore omitted this from our final method. \n\nOverall, driven by our “magnification loss” we are able to handle occlusions and disocclusions adeptly as evidenced by our qualitative results. We will revise our manuscript to include a discussion of this point.\n\n**SSIM**\n\nThere are two main advantages that DeepMag has when evaluated on SSIM. Firstly, DeepMag is trained with a loss that is relatively similar to SSIM. As a result it tends to score higher on this metric. Secondly, the SSIM metric is computed on the synthetic test set from DeepMag. This test set is generated in a procedure that is very similar to how the train set for DeepMag was generated, and is therefore quite in-domain for the DeepMag model. On the other hand, our model was trained on real videos and the test set is therefore significantly out-of-domain both in terms of content (derived from COCO and PASCAL) and motion (only translations). We note that despite this domain gap we are able to outperform DeepMag on the Motion Error metrics on the synthetic test sets and achieve good SSIM results.\n\n**Moving Background in Supplemental Video**\n\nWe believe that you are correct that the shimmering background comes from optical flow errors. The background of the cat video, which is extremely out-of-focus and features bokeh artifacts, is out-of-distribution and therefore particularly hard for the optical flow model. Our model is closely tied to the performance of the underlying optical flow model, and we will add a discussion of this in our limitations section.\n\n**Adaptive Magnification Loss**\n\nYour suggestion of only applying the magnification loss to segmented objects should work in removing background motion, and would certainly be an interesting extension of our loss for future work. It would however also add complexity to the training process as it would be necessary to generate segmentations and make foreground-background predictions. In addition it could remove motion that the user might want to keep. Therefore for this paper we opt to keep our method simple and just train to magnify all motion, and then allow the user to choose at inference time which areas of the video to magnify through our targeting procedure. \n\n**GAN Loss**\n\nA GAN loss could certainly help the performance of the method. However, we wanted to focus this work specifically on the performance of the self-supervised loss that we introduce. To this end we opted to omit a GAN loss as it would make it hard to disentangle the roles of the adversarial loss and the motion magnification loss. Even without a GAN loss we are able to produce magnified sequences that are competitive with or outperform existing supervised methods, demonstrating the effectiveness of our proposed method.\n\n**Dataset Information**\n\nWe are more than happy to release information about the dataset upon acceptance. In fact, enough data to perfectly reconstruct the dataset should already be in the paper and the supplemental material. Table 1 contains the temporal strides used to sample frame pairs from the 5 constituent datasets, in addition to the number of frame pairs before and after motion filtering. Appendix A3 contains additional information about how the constituent datasets were sampled, details on how the motion filtering was performed, and an overview of how the test set was constructed. We plan on releasing our datasets and all of our code upon acceptance, including the code used to sample, filter, and compile our train and test datasets.\n\n**Limitations of Off-the-Shelf Models**\n\nWe agree with the observation that our method is fundamentally limited by the quality of off-the-shelf models, and will add this to our limitations section. We however also like to caveat that this dependence on off-the-shelf models means that our method can also benefit from improvements in optical flow models.\n\n&nbsp;\n\nThank you for your insightful review, and please let us know if you have further questions or need further clarification."}, {"review_id": "fxxe6vxg3J", "reviewer": "Reviewer_hFGm", "summary": "This paper proposes a self-supervised model to solve the Lagrangian motion magnification problem without needing ground-truth labels. The network takes as input the two input frames and a magnification factor that ranges from 1 to 16, and outputs a generated frame that has magnified motion from the first frame. Off-the-shelf optical flow networks are used in loss computation for self-supervision. Test-time adaption has been explored to enhance the generation quality. Experiments show promising results.\n", "questions": "1. As you included the optical flow network in the whole computation graph with gradient computations, did you freeze its weights to make sure it does not shift away? If so, please state so explicitly in the paper.\n2. Line 185-186: Why is the positional encoding conditioned on the factor $\\alpha$? How do you do it? It does not make sense to me if no explanations are given.\n3. Test-time adaptation: this trick adds in overhead on inference time. How efficient is it?\n4. Did you tackle the occlusion issue? I believe this is an issue both for new frame generation and optical flow estimation. A simple warping using optical flow may be good enough if there are no occlusions at all.\n\nAdditional comments:\n1. Line 26-28:  \"The datasets and learning procedures that are used by these models are designed to be general-purpose, with a particular focus on ensuring that they apply to a variety of motions, objects, and scenes\". Your cited methods are all supervised methods, which are usually trained on large synthetic datasets that could be totally different from real use cases. The generalization ability of these models is still an open question. I think a better idea is to weaken this statement.\n2. Line 32-34: \"And generate a new image pair whose predicted optical flow is $\\alpha$ times as large as that of the input\". Maybe add that the new image pair should share the same reference frame from the input, to avoid confusions.\n3. Line 33-34: add $(\\alpha \\geq 1)$ to be more clear. Your method also makes sense even if $\\alpha < 1$, so it is better to clarify your work cases.\n4. Line 38: \"it\" -> \"our method\".\n5. Line 61: need a citation for \"Eulerian approaches\".\n6. Line 69: \"it uses it\" -> \"we use it\".\n7. Line 115: It is better to make the symbol $x$ bold like $\\mathbf x$ since it is a vector.\n8. Eq 1: Maybe add that this equation assumes no occlusions.\n9. Fig 6 caption: \"subset\" -> \"subsets\".\n10. Repeated references: [21] and [22], [37] and [38].", "limitations": "Maybe need to add occlusions as a key limitation", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Good writing; overall clear. The studied problem has received arguably less attention in the research community, but the related work section is detailed and well-structured, which especially helps the readers to catch up.\n2. The method is very simple and easy to understand.\n3. Experiments show promising results.\n4. The authors promised to release full code upon acceptance.", "weaknesses": "1. The targeting application of this task is not clear. Why is this task important? Which data domains or scenarios do we want it to work? If our goal is just to detect small motions, we can develop optical flow estimation methods that work specifically for small motions. Even for existing state-of-the-art optical flow networks, detecting small motions is generally not a big issue, and it should not be hard to find a way to visualize small optical flow. Why do we need to generate a video? Maybe adding some application examples in the introduction and some results on related datasets will help the reader better understand the background and goal of this task.\n2. There are still some confusions on the method. See questions below. \n3. Some minor edits. See additional comments below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful review. We would like to address your questions as follows.\n\n**Importance of Motion Magnification**\n\nMotion magnification has applications in fields ranging from medical imaging to structural engineering to micro-expression analysis. Motion magnification enables us to amplify subtle movements thereby revealing dynamics that may have previously been hidden or hard to interpret. In the realm of medical imaging motion magnification may be used to magnify minuscule motions within tissues or organs to assist medical experts for diagnosis. For engineers motion magnification may serve as a powerful tool for detecting structural faults or weaknesses in buildings, bridges, or machinery. Motion magnification can also be applied to emphasize emotions, actions, or facial expressions that would otherwise be too small to detect.\n\n**Importance of Magnification Targeting**\n\nTo magnify the motion of certain objects instead of motion in the entire frame, previous phase-based work used temporal filtering to select motion within specified frequency bands. This is a rather unintuitive interface to select objects for magnification and is used mainly because frequency filtering is particularly natural for phase-based methods. In addition, there may be multiple objects in a video moving at similar frequencies making it hard to isolate each object individually.\n\nWe propose “targeted magnification” as a natural and intuitive way to allow users to magnify  objects within a segmented area. With recent advances in object segmentation it is much easier and straightforward for users to click and choose to magnify specific objects, giving precise control over what areas of the video to magnify.\n\n**Directly Using Optical Flow for Motion Detection**\n\nVisualization of optical flow is generally hard to decipher and motion magnification offers a much better alternative. Colorwheel visualizations where directions are mapped to arbitrary colors can be unintuitive to understand, especially when the motion changes quickly and many colors flash by rapidly. In addition, the magnitude of the motion is typically encoded by the saturation and is thus hard for humans to make fine differentiations between flow magnitudes. \n\nOverall, motion magnification is an elegant solution to the problem of detecting, visualizing, and understanding small motions in videos. We will include a discussion of these points in our paper. Please let us know if you have further questions.\n\n**Frozen Optical Flow Network**\n\nYes, the optical flow network is frozen but is still backpropagated through to give a learning signal to the generation network. We will make this explicit in the paper.\n\n**Alpha Encoding**\n\nIn order to pass the desired magnification factor, $\\alpha$, to our network we use a \"positional embedding\" scheme to encode the scalar $\\alpha$ into a vector. The details of this implementation can be found in Section A2 of the appendix as well as in the code from the supplementary materials. In particular, we use a standard sinusoidal embedding scheme where alpha is passed through a series of sinusoids with exponentially increasing wavelength and we stack the result into a vector. We apologize for any confusion and will edit the text in our paper so that this is clearer.\n\n**Test Time Adaptation Efficiency**\n\nThe efficiency of the test time adaptation experiments presented in the paper were not specifically optimized, nor was their performance measured. We rerun the experiments corresponding to the three examples from Figure 2 in the paper (\"baby,\" \"cats,\" and \"pole\") to provide more careful measurements of test time adaptation efficiency. Qualitative visualizations of improvement and quantitative measures of efficiency are in the attached PDF in Figure 1 and Table 1. We find that even with a thousand gradient steps and tens of minutes, improvement can be seen on out of distribution data. We give a precise explanation of the test time adaptation procedure below, which will be added to the manuscript.\n\nGiven a video for test time adaptation we construct a dataset using every frame, with minor rotation and color augmentation. We then train for N epochs, where an epoch consists of all the frames in the video. The learning rate is set to be 1e-4 (a third the learning rate during train time) and the batch size is 8. Note that the training times vary based on the number of frames and the size of the video.\n\n**Occlusion and Disocclusion**\n\nThis is an excellent observation. As can be seen in the supplemental video our model produces qualitatively good results even in the presence of occlusions and disocclusions. This is because optical flow models are already designed to handle occlusions and disocclusions well, so our magnification loss also incentivizes correct predictions in these scenarios. \n\nThe other component in our loss, the color loss, can be affected by occlusions and disocclusions. To mitigate this, in early experiments we tried masking occlusions with our loss, where we check for occlusions by computing flow both forwards and backwards and performing a cycle consistency check. We apply this occlusion mask to the color loss such that pixels that are occluded or disoccluded do not contribute to the loss. We found that results did not change significantly with occlusion masking, possibly because the magnification loss already handled occlusions and disocclusions adequately, and we therefore omitted this from our final method. \n\nOverall, driven by our “magnification loss” we are able to handle occlusions and disocclusions adeptly as evidenced by our qualitative results. We will revise our manuscript to include a discussion of this point.\n\n**Typos and Suggestions**\n\nThank you for findings these. We will revise our paper to include your suggestions and fix typos.\n\n&nbsp;\n\nThank you for your thoughtful review, and please let us know if you have further questions or need further clarification."}, {"review_id": "RFDWJw5D9j", "reviewer": "Reviewer_57E8", "summary": "The paper proposed Lagrangian motion magnification using the pre-trained optical flow network. Magnification loss induces that the optical flow of the magnified frame matches with the optical flow of a given frame by $\\alpha$ times; color loss regularizes the color consistency between a given and magnified frames. Test-time adaptation improves the quality of magnification on out-of-domain. Experiments show competitive performance compared to the prior arts in terms of SSIM and the proposed evaluation metrics.", "questions": "My main concerns are the used term and fair comparison. See the weaknesses part.\n\n=====\n\nI update my rate from 4 to 6 because the authors will reflect the discussion below.", "limitations": "Limitations and broader impacts are described at the end of the main paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. Simple and effective algorithm to train motion magnification network using the off-the-shelf optical flow network.\n2. Given the off-the-shelf optical flow network, this approach enables the training on large-scale unlabeled videos.\n3. Targeted magnification and test-time adaptation might provide a better user experience.\n4. The proposed method seems to be independent of the architecture of the neural network.", "weaknesses": "1. The term should be used carefully. I am not sure that the proposed method can be named \"self-supervised\" because the off-the-shelf optical flow network, which is used in experiments, is trained by supervised learning. If authors want to use the term \"self-supervised\", the self-supervised optical flow network should be used in the main experiments and the supervised one would be the strong baseline to be compared; It is not sufficient that the self-supervised optical flow network can be used in theory. \n2. Evaluation metric is limited. To justify the underperformance in SSIM, the authors notice this phenomenon in the last sentence in Table 3, \"DeepMag explicitly trains for SSIM\". The proposed algorithm is also optimized by the proposed evaluation metric, Motion Error. For the same reason, I cannot be convinced about the quantitative results. \n3. It is better to include the limitation to use the optical flow network. The optical flow network is inferior to estimate the subtle motion. It is related to the underperformance in the 0.04px subpixel test of Table 3. Thus, the limitation induced by the optical flow network should be investigated as an ablation study because this subtle motion is important in magnification.\n4. I think that this proposed method is more general than supervised or self-supervised learning because this is determined by which optical flow network is used. How about using the synthetic data together?\n5. Is the network architecture different from DeepMag? As for the control experiments, do the number of parameters affect the performance directly? \n6. I think that DeepMag is sufficient as the strong baseline. However, I wonder why Warp Nearest and Bilinear are used, and the more advanced hand-crafted algorithms [A, B] are not used as baselines. It is because evaluation data might contain large motion?\n\n[A] Phase-Based Video Motion Processing (SIGGRAPH 2013)\n\n[B] Riesz Pyramids for Fast Phase-Based Video Magnification (CVPR 2014)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and comprehensive feedback. We would like to address your points below:\n\n**The Term “Self-Supervised”**\n\nWe term our method “self-supervised” because it does not need ground truth magnified motions, as opposed to previous work DeepMag. It is common practice in the ML and vision communities to label methods self-supervised even if they use supervised components somewhere in the system, particularly optical flow. Below are papers presented at reputable conferences that follow this standard:\n\n- \"Self-Supervised Learning of Motion Capture.\" H. Fish Tung, H. Tung, E. Yumer, K. Fragkiadaki. NeurIPS 2017.\n    - Self-supervised objective for predicting a 3D body mesh from a video, with a term that uses optical flow computed using FlowNet2.0.\n- \"The Sounds of Motion.\" H. Zhao, C. Gan, W. Ma, A. Torralba. ICCV 2019.\n    - Self-supervised method for sound source separation with motion as auxiliary information, extracted from a pretrained PWCNet.\n- \"Self-Supervised Learning via Conditional Motion Propagation.\" X. Zhan, X. Pan, Z. Liu, D. Lin, C. Loy. CVPR 2019.\n    - Self-supervised pretext task in which a network is required to predict optical flow from sparse motion signals, both computed from the supervised LiteFlowNet.\n- \"Self-Supervised Learning of Audio-Visual Objects from Video.\" T. Afouras, A. Owens, J. Chung, A. Zisserman. ECCV 2020.\n    - Self-supervised loss to find \"audio-visual objects,\" which utilizes tracks computed using PWC-Net.\n- \"Self-Supervised Representation Learning from Flow Equivariance.\" Y. Xiong, M. Ren, W. Zeng, R. Urtasun. ICCV 2021.\n    - Learns representations by enforcing equivariance to a warping under optical flow, computed from RAFT.\n\nIn addition, we train a version of our model using ARFlow, a fully unsupervised optical flow model, and present results in the attached PDF in Tables 2 and 3. Despite never fully finishing training, it performs well on metrics and beats DeepMag a significant amount of the time. We will include completed results in our manuscript.\n\n**Evaluation Metrics**\n\nWe would be curious if the reviewer has an alternative metric in mind. Motion magnification is an incredibly hard task to measure quantitatively with no standard benchmarks, but with many possible metrics with different trade-offs. As such, we aim to present a holistic, fair, and well-rounded view of the performance of our models and baselines.\n\nTo do this we use both SSIM, which favors DeepMag and is used in their evaluations, and we introduce the Motion Error metric, which favors our method. We mitigate the advantage our model by using *three* optical flow models to compute the Motion Error: RAFT which is used during our training, and PWC-Net and GMFlow which is never used by our method at all. Our method outperforms DeepMag on all instances.\n\nIn addition, DeepMag has a significant advantage in the synthetic evaluations as it is being tested on in-domain data. This is because the process used to generate the synthetic train and test set are almost identical (both use PASCAL VOC objects on COCO backgrounds undergoing only translations). Despite this advantage we are still able to outperform DeepMag on the the Motion Error metrics and perform well on the SSIM metrics.\n\n**Very Small Flows**\n\nWe agree that our method is closely tied to the performance of the optical flow method used during training and will add more discussion of this in the limitations section. On the other hand, because our method does not rely on a specific optical flow method it is free to take advantage of future progress in optical flow models.\n\nAs for ablations on small motions, we point out that we already investigate small sub-pixel motions in depth in table A3 of the appendix where our method outperforms DeepMag on motion error. If there are specific ablations that you believe would be helpful please let us know.\n\n**Joint Supervised and Self-Supervised Training**\n\nOur method can certainly be combined with existing supervised approaches. However, our primary goal in this paper was to investigate our motion magnification objective. For this reason we opted to focus on the methods in isolation as combining them could make evaluation difficult. It would not be clear how to disentangle the benefits from different objectives.\n\n**Architectural Differences**\n\nYes, the architectures are different. Because our method is not tied to a specific architecture we use the simple and ubiquitous UNet. DeepMag uses a bespoke architecture designed specifically for motion magnification, consisting of an encoder, a decoder, and a manipulator. The encoder encodes frames into a \"texture representation\" and a \"shape representation\" and the manipulator modifies the shape representation to achieve motion magnification.\n\nIn addition, we show results in the attached PDF of a smaller model in Tables 2 and 3. This model has 1.09 million parameters compared to DeepMag’s 0.92 million. Due to time constraints we were only able to fully train the model, but despite this it performs similarly to our original model as compared to DeepMag. We will add completed results to the paper. \n\n**Forward Warp Baselines**\n\nThe forward warp baselines we use are the simplest Lagrangian motion magnification methods. Therefore these methods serve as a point of reference, or a lower bound, for the reader. In addition, they give a sense of how well using *just* optical flow can perform, as opposed to using it in our proposed objective.\n\n**Phase Based Baselines**\n\nDeepMag shows superior performance against the phase-based methods so we believed it sufficient to compare against DeepMag. Moreover, evaluation of phase-based methods typically requires application of a temporal filter which makes it quite tricky to design a fair comparison. For an example of this please see the DeepMag paper.\n\n&nbsp;\n\nThank you for your thorough review, and please let us know if we can further answer any questions you may have."}]

Author rebuttal:
We thank all the reviewers for their insightful, thorough, and constructive reviews which have helped to improve our manuscript greatly. Please find individual responses to your questions and comments below. We look forward to the discussion period.

Meta-review: While the initial reviews were mixed, this submission eventually received unanimously positive reviews from the reviewers after the discussion period. The AC agrees and recommends acceptance. The authors are encouraged to incorporate the feedback in the camera-ready version.
Decision: Accept (poster)