Title: Self-Supervised Motion Magnification by Backpropagating Through Optical Flow
Venue: NeurIPS 2023
Accepted: yes
Keywords: Video Processing

Motion Processing

Motion Magnification

Optical Flow
Abstract: This paper presents a simple, self-supervised method for magnifying subtle motions in video: given an input video and a magnification factor, we manipulate the video such that its new optical flow is scaled by the desired amount. To train our model, we propose a loss function that estimates the optical flow of the generated video and penalizes how far if deviates from the given magnification factor. Thus, training involves differentiating through a pretrained optical flow network. Since our model is self-supervised, we can further improve its performance through test-time adaptation, by finetuning it on the input video. It can also be easily extended to magnify the motions of only user-selected objects. Our approach avoids the need for synthetic magnification datasets that have been used to train prior learning-based approaches. Instead, it leverages the existing capabilities of off-the-shelf motion estimators. We demonstrate the effectiveness of our method through evaluations of both visual quality and quantitative metrics on a range of real-world and synthetic videos, and we show our method works for both supervised and unsupervised optical flow methods.

Self-Supervised Motion Magnification by Backpropagating Through Optical Flow

Abstract

This paper presents a simple, self-supervised method for magnifying subtle motions in video: given an input video and a magnification factor, we manipulate the video such that its new optical flow is scaled by the desired amount. To train our model, we propose a loss function that estimates the optical flow of the generated video and penalizes how far if deviates from the given magnification factor. Thus, training involves differentiating through a pretrained optical flow network. Since our model is self-supervised, we can further improve its performance through test-time adaptation, by finetuning it on the input video. It can also be easily extended to magnify the motions of only user-selected objects. Our approach avoids the need for synthetic magnification datasets that have been used to train prior learning-based approaches. Instead, it leverages the existing capabilities of off-the-shelf motion estimators. We demonstrate the effectiveness of our method through evaluations of both visual quality and quantitative metrics on a range of real-world and synthetic videos, and we show our method works for both supervised and unsupervised optical flow methods.

1 Introduction

Motion magnification methods [32, 58, 40] increase the size of tiny motions in a video, revealing subtle details that are difficult to discern with the naked eye. However, existing methods come with significant limitations. Early hand-crafted methods generally require small periodic motions [64, 58] or a human in the loop [32]. More recent supervised learning methods [40] require ground-truth training examples, such as videos before and after magnification, which are difficult to obtain without synthetic data. Creating this synthetic data is a challenging problem, since it seemingly requires capturing all of the possible objects and motions that one might ever want to magnify.

In parallel, the field of motion estimation has addressed many closely related challenges. Modern optical flow networks [65, 54, 51, 10] are designed to track objects undergoing complex motions, both large and small. Most of these methods are trained on supervised datasets that capture a wide variety of objects, but parallel work has shown the effectiveness of unsupervised flow estimation [33, 27]. We ask whether we can use motion estimation models to train magnification models, taking advantage of their existing capabilities and reducing the need for special-purpose training data.

We propose a simple motion magnification model whose supervision signal comes from an offthe-shelf motion estimation model. Our method exploits the fact that optical flow networks are differentiable, and thus can be used as part of a loss function. We train a model to take a pair of video frames and a magnification factor α as input, and to generate a new pair whose predicted optical flow is α times as large as that of the input. We simultaneously optimize a regularization loss that preserves the visual appearance of each tracked pixel in the generated video. Notably, our model can

(b) Test-time Adaptation

(iv) Original

(v) No TTA

(vi) TTA

(i) Original Frame

(iii) TTA Frame

(ii) No-TTA Frame

(a) Magnification

(i) Original Frame

(ii) Left Fork

(iii) Right Fork

(iv) Original

(v) No Targeting

(vii) Magnifying Right Fork

(c) Targeted Magnification

(i) Original

(ii) Magnified

(i) Original

(ii) Magnified

𝑦𝑦

𝑡𝑡

be trained solely using real unlabeled videos. The optical flow estimator is only used within the loss function during training-we do not require it at test time.

Our model's simplicity and its ability to be trained solely through self-supervision provides it with several advantages over other approaches. We show that we can improve our generated videos through test-time adaptation [52, 26] by finetuning on a given input video. Our formulation can also be easily extended to magnify the motions of a single object, specified via a user-provided mask. In addition, we show results using both off-the-shelf supervised [54] and unsupervised [33] optical flow methods.

Our method is closely related to Lagrangian magnification methods [32, 19], which tracks individual pixels through a video and then amplifies their motion. While this approach is intuitively appealing and was the basis for the earliest magnification methods, a major shortcoming is that it is not clear how to combine the tracking and synthesis together. Previous Lagrangian approaches [32] often rely on hand-crafted pipelines that combine motion estimation, warping, and inpainting steps. By contrast, our method avoids these pitfalls by using off-the-shelf image-to-image translation architectures [45], and uses tracking only within the loss function.

We demonstrate the effectiveness and flexibility of our model in several ways. First, we show through experiments on real and synthetic videos that the optical flow in the generated videos more closely matches the desired motion than videos generated by baseline approaches, such as methods based on warping or supervised learning [40]. Second, we show qualitatively that our model can successfully magnify motions for a variety of videos, containing both small and large motions. Finally, we show that we can improve generation quality using test-time adaptation, and magnify individual objects using user-supplied segmentation maps.

2 Related Work

Lagrangian magnification. Motion magnification methods can be broadly categorized as either Lagrangian [32] or Eulerian [64], a classification borrowed from fluid dynamics. Lagrangian methods explicitly track pixels, then generate new frames by forward warping (i.e., splatting) using magnified velocity estimates. Liu et al. [32] originally proposed the motion magnification problem and solved it using a Lagrangian approach. They modeled motion as feature trajectories and (with humanin-the-loop assistance) cluster pixels by similarity in position, color, and motion. Recent methods have extended Lagrangian magnification with more accurate optical flow [17]. While our approach performs explicit tracking and thus is closely related to Lagrangian methods, we decouple generation from tracking: we produce images using off-the-shelf networks and track only within the loss function . This allows us to avoid some of the challenges of the 'warp and inpaint' approach, such as handling occlusions and filling holes.

Eulerian magnification. Eulerian methods magnify motions without explicit motion estimation. Instead, they generate a magnified video by amplifing the temporal changes at fixed locations/pixels. Wu et al. [64] decomposed a video into frequency bands and applied temporal filters to extract a signal at a specific bandpass. The extracted signal is then amplified by a magnification factor and added back to the video. Wadhwa et al. [58] proposed a phase-based method, using complex steerable pyramids [47] to decompose the video and separate the phase from the amplitude to amplify the temporally-bandpassed phases. Later work improves efficiency using Riesz pyramids [59] and removes large motion by decomposing a scene into foreground and background layers [11]. Zhang et al. [72] magnified small motions while ignoring large motions by amplifying the motion field acceleration using second-order temporal filters. Concurrent work [15] extended 2D Eulerian methods to 3D motion magnification. Eulerian methods are well-suited to tracking small motions at high spatial frequencies, and may struggle when handling large motions [58], whereas Lagrangian methods' magnification quality is determined by the quality of the optical flow predictions [64]. Since our method's loss function is defined using optical flow, its capabilities are more similar to those of Lagrangian methods.

Learning-based magnification. Several recent works have proposed learning-based motion magnification models. Oh et al. [40] created a synthetic dataset containing image segments extracted from PASCAL VOC [12] as moving foreground and images from COCO [30] as background, and trained a model to regress a ground truth magnified image from two video frames. Inspired by the steerable pyramid used in Eulerian methods [58], they also proposed an architecture that has inductive biases that encourage it to generate crisp images akin to those in Eulerian magnification, by decomposing the image into a shape and texture representation. Other work has extended this supervised magnification approach using 3D CNNs [28] and lightweight architectures [48], and has magnified microexpressions using attention-based models [63]. In contrast to these approaches, our method is self-supervised and learns from unlabeled videos. Recently, Gao et al. [19] use a very similar supervised learning objective and dataset but augment the model of Oh et al. [40] with inductive biases that encourage Lagrangian-like magnification by adding an attention map that is guided by an optical flow field. Instead of making optical flow part of our architecture, as an inductive bias that aids supervised training, we use it to define a self-supervised loss function .

Motion estimation. The field of motion estimation is closely related to motion magnification. Early work solved linearized models after making color constancy assumptions [35] or solved smoothness-regularized models with variational methods [23]. Later work added robust losses and inference strategies [49, 3, 31, 14]. While these methods are well-suited to the subtle motions considered in motion magnification, it is difficult to use them as part of a loss function (as we do in this work), since it is not straightforward to differentiate through during gradient-based learning. Recent methods based on neural networks address this issue, since they are differentiable and highly accurate. A variety of recent methods train neural networks using real or synthetic optical flow data [16, 25, 43, 24, 67, 51, 65] or unlabeled video data [71, 44, 61, 34, 27, 2]. Our use of a pretrained flow model removes the need for special-purpose supervised motion magnification datasets.

Optical flow within a loss function. We take inspiration from recent work that uses differentiable optical flow models to define loss functions that internally perform motion reasoning. Geng et al. [20] used flow to obtain robustness to small positional errors on image generation tasks. Goyal et al. [21] used flow to measure the distance to a goal state for robotic planning. Other work has backpropagated through optical flow as part of a pose estimation model [55]. By contrast, we use flow to compare the motions in two videos.

3 Method

We describe traditional Lagrangian motion magnification, and our proposed self-supervised Lagrangian magnification model.

3.1 Lagrangian Motion Magnification Overview

Lagrangian motion magnification methods magnify motion by tracking pixels over time, then resynthesizing the video such that the motion of each pixel has increased by a desired amount. More concretely, a point x in the initial frame I 0 of a video might be displaced by motion field T ( x ; t ) in

<latexi sh

1_b

64="MOZ

T

HYc

87W

k

G

S

Jr9g

>A

B

n

VDL

N

E

y

fU

oP

K

2

Q

m

5

d

C

X

v

R

0

uj

+

z

3

q

w

I

/

<latexi sh

1_b

64="3MUkK

Co

AzY5

0

O

B

cuGr2

>

7X

VDLSgN

E

y

f

9

P

Q

8

I

HJ

n

m

Z

/

wdFv jqp

T

W

R

+

<latexi sh

1_b

64="XI

uTDy

Uc

OS

Y5QPV

>A

B

n

L

gN

E

r

f

9

o

K

2

m

k

dzCZ

8

v

R70G

H

W/

M

q

3

j

p

F

w

+

J

<latexi sh

1_b

64="U5NgBVzIfpRX

+

d

cOG

j

E

>A

8n

DLS

M

F

3

2vqk

w

K

P

J

y

C0

u/

7

9

T

o

r

W

Z

Q

m

H

Y

<latexi sh

1_b

64="F9pKU

dY8

M

7/

PRO

Sq53

D

>A

B

n

c

V

L

N

J

2v

k

g0

w

y

+

zI

ZG

Efo

W

T

C

u

r

X

m

Q

j

H

<latexi sh

1_b

64="MGSDOFu3

d

X

>

C

H

cnV

N

EPU

BQ

w

J

Ro

I

Wy

r

2

9

7

Z

T

0

L

Y

K

g

p

<latexi sh

1_b

64="EqA3S

XwkR

2

I

v

o

j

g

>

CT

c

ZHL

y

FMYz

7r

/

dB

P

u

G

W

KU

m

9

pn

J

5

f

V

Q

N

0

8

O

<latexi sh

1_b

64="EdpFq7SZ

CO

PB5

zr

m

WI

>A

/n

c

VDL

N

J3U

2v yM

8

w

0

k

g

Yo uHG

T

Q

9

X

K

R

j

+

f

<latexi sh

1_b

64="Cm

j

d

ZR0KQ8w

VXv9

2

YS

>A

B

c

DJ

gN

EO

W

M

Go

u

p

L

P

k

U/

F

+

f

3

5

q

r

7

T

H

z

I

y

n

U-Net

I

0

I

t

α

˜

I

t

L

color

=

1

|P|

∑

P

‖

-

‖

1

L

mag

=

∥

∥

∥

∥

∥

∥

α

·

-

·

∥

∥

∥

∥

∥

∥

1

F

(

I

0

, I

t

)

F

(

I

0

,

˜

I

t

)

(a) Architecture

(b) Losses

correspondences

from

F

frame t . Assuming color constancy, we have:

where I t is the frame at time t and I ( · ) indicates pixel access. To generate a version of the video magnified by a factor α 1 , one could synthesize a new frame ˜ I t by increasing the distance traversed by each pixel:

For example, early work in Lagrangian motion magnification [32] aimed to achieve this by performing forward warping (splatting). However, this leads to holes and aliasing artifacts [53]. Moreover, there is no mechanism for dealing with occlusions: after warping, background pixels may move in front of the foreground. Finally, such a model would not be able to deal with appearance changes without additional constraints (e.g., due to lighting variation).

3.2 Self-Supervised Lagrangian Magnification

We propose a motion magnification model that avoids the limitations of previous Lagrangian methods, and that can be trained solely using unlabeled video (Figure 2). Consider F ( I 0 , I t ) , the motion field obtained by computing optical flow between a reference frame I 0 and another frame I t . Our goal is to generate a magnified frame ˜ I t such that its predicted flow is α F ( I 0 , I t ) , where α is the magnification factor. To do this, we optimize an objective to minimize the difference between the estimated flow F ( I 0 , ˜ I t ) and the desired flow:

which we term the magnification loss . When F is implemented as a neural network, we can optimize this loss using gradient-based learning methods.

Optical flow models are invariant to a variety of photometric changes (e.g., changes in illumination). Thus, optimizing Eq. 3 alone would result in an underconstrained problem. To ensure that the generated frames match the colors in the input video and to also regularize our problem, we use an additional color loss . This loss ensures that corresponding pixels in generated frame ˜ I t and original frame I t are the same color. To enforce this, we put both frame I t and ˜ I t into correspondence with the common reference frame I 0 by backward warping each frame with their respective flows: F ( I 0 , I t ) and F ( I 0 , ˜ I t ) . We then measure the distance between the warped frames. This results in a loss:

This loss also disincentivizes adversarial examples against the flow network. Our full loss is a weighted sum of these two losses:

One benefit of putting optical flow into the loss function is that our model needs access to the flow network only during training; at inference time the model simply takes in two frames and outputs a magnified frame. Pseudocode for our method for training and inference can be found in Algorithm 1 and Algorithm 2 respectively.

3.3 Image Generation Architecture

One benefit of our formulation is that our proposed loss (Eq. 5) is independent of the image generation architecture, so that 'off-the-shelf' image translation architectures can be used to create a magnified image. This is in contrast to other learning-based methods, which incorporate inductive biases within the network to generate images that closely resemble the input (e.g., by altering the structure of an image while preserving the texture [40]). To generate magnified frames, we use a standard U-Net architecture [45] that takes two input frames (the reference frame and the frame to be magnified) concatenated channel-wise. We encode magnification factor α using a sinusoidal positional embedding [39] and tile it to match the same spatial dimensions as the input frames. We concatenate this embedding channel-wise to the input frames. Please see Section A2 in the appendix for full architectural details.

3.4 Targeted Magnification

Our model gives us the ability to vary the magnification factor spatially within an image. We achieve this by providing different values of α for each pixel at inference time. This works even when training with spatially constant alpha maps, due to the fully convolutional nature of our network 2 . As a special case, we magnify the motion of a single object by setting the magnification factor to a value of α within an object segment, and to 1 everywhere else. In practice, we use the recent Segment Anything Model (SAM) [29] to extract a mask for a given object in the reference frame and then dilate the segmentation mask by a small fixed number of pixels. We provide pseudocode for targeted magnification during inference in Algorithm 2.

3.5 Test-Time Adaptation

Since our model is entirely self-supervised, we can finetune it at inference time on the input video [52, 26, 18]. This allows us to adapt to new motions and content. We find that this can significantly improve results, especially for out-of-domain videos. Previous supervised approaches do not have this capability as they require labeled data. To perform test-time adaptation we take the frames in a given inference video, apply minor cropping and color augmentations, and finetune the model with our loss.

4 Experiments

We evaluate our model both quantitatively through experiments with real and synthetic data, and qualitatively on real videos. We highly encourage the reader to view our website, since the magnified motions can be challenging to visualize in static images.

4.1 Implementation Details

We train two variations of our model. One that uses ARFlow [33], and one that uses RAFT [54] as the optical flow network. Since ARFlow is self-supervised, using it results in a fully self-supervised motion magnification method. RAFT on the other hand is a supervised flow model with very good performance, giving us a weakly-supervised magnification method that serves as a rough upper bound to the performance of a fully self-supervised method. More discussion can be found in Section A2 in the appendix. All qualitative results presented in this paper are from the RAFT model, except in Figure 6 in which we compare the ARFlow and RAFT models.

Because motion magnification is multiplicative, we sample random magnification factors α exponentially, such that log 2 ( α ) ∼ U (log 2 ( α min ) , log 2 ( α max )) , where α min = 1 and α max = 16 . For larger magnification factors that exceed α max, we find that recursive application of our model produces high-quality predictions. More implementation details are available in Section A2 in the appendix.

4.2 Dataset

Because our method is self-supervised, we benefit from a large, diverse dataset of videos. To this end we curate a dataset containing 145k unlabeled frame pairs from several existing datasets, including YouTube-VOS-2019 [68], DAVIS [42], Vimeo-90k [66], Tracking Any Object (TAO) [8], and Unidentified Video Objects (UVO) [60]. We remove frame pairs with large motions (e.g., from the camera or objects), since these frames are less likely to be used in magnification appliactions. We also remove frame pairs that are near-identical by setting a lower bound on the MSE between the two frames. In addition to a training set, we collect a test set consisting 650 frame pairs for evaluation, which we refer to as real-world test set . We provide more details of dataset collection and filtering in Section A3 of the appendix.

For testing, we also use the synthetic test set from Oh et al. [40], which is generated by compositing objects from PASCAL VOC [12] to backgrounds from COCO [30] at varying levels of subpixel motion and noise. We refer to this test set as the synthetic test set . Finally, we qualitatively assess our method and baselines on various real-world videos , which vary greatly in subject matter, motion complexity, and lighting conditions. These videos include de-facto standard benchmarks for motion magnification used in previous works [64, 58, 59, 5, 40], as well as new videos.

4.3 Metrics and Baselines

Metrics. Similar to Oh et al. [40], we use SSIM [62] as an evaluation metric. However, because SSIM requires ground-truth magnified images, we can only use this metric to evaluate models on the synthetic test set. In order to evaluate our method on the real-world test set, we propose another metric for motion magnification that does not require ground truth, which we term motion error . Inspired by the accuracy and robustness of recent optical flow methods, we assume that the flow estimate from an optical flow method is ground truth and calculate an end-point error between the predicted flow and the desired magnified flow.

This metric is identical to L mag if the same optical flow model is used for evaluation as is used during training. In order to ensure a robust metric, we calculate the motion error metric using a wide range of optical flow networks, including PWC-Net, GMFlow, and RAFT, that have been trained on various datasets. We set the number of iterations for RAFT to 20 during evaluation, while keeping it at 5 during training. This serves the dual purpose of allowing us to train more efficiently, but also evaluate more equitably.

In addition, we compute the per-pixel ratio between the flow magnitudes of the unmagnified and magnified frames, and calculate their average deviation from the desired magnification factor α . This metric, which we refer to as magnification error , was not explicitly trained for and serves as another indicator of magnification quality.

(i) Original

(ii) Warp Nearest

(iv) Oh

(v) Ours

(vi) Ours (TTA)

(a) Baby

(i) Original

(ii) Warp Nearest

(iv) Oh

(v) Ours

(vi) Ours (TTA)

(b) Cats

(i) Original

(ii) Warp Nearest

(iv) Oh

(v) Ours

(vi) Ours (TTA)

(c) Pole

Ours

Oh

Input (

Train

)

Oh

Ours

Input (

Camel

)

Input (

)

Oh

Ours

Baselines. We compare primarily against the method of Oh et al. [40], a neural network based approach trained on a synthetic dataset. Additionally, we implement forward warp baselines that use nearest and bilinear sampling to warp pixels according to the amplified flow, along with a nonparametric inpainting method [41], which we term Warp Nearest and Warp Bilinear respectively. We also compare against FLAVR [28], a 3D U-Net trained for frame interpolation and finetuned on the same synthetic dataset as Oh et al. , albeit at a constant magnification factor of α = 10 . Finally, we provide qualitative comparisons in Section A5 of the appendix to Neural Implicit Video Representations (NIVR) [36], a method that fits an implicit representation to a video and displays emergent motion manipulation behavior.

4.4 Comparison with the State-of-the-Art

Visual quality. We show qualitative results on real-world videos. In Figure 4 we show still frames and closeups of results from Oh et al. and our method. And in Figure 3 we plot y -t slices through the video volume, with one dimension being time and the other being spatial 3 .

One qualitative finding is that while the flow network may be noisy on a specific video, our model trained with the same flow network can be much more robust. This can be seen clearly in the baby sequence of Figure 3. The forward warp methods, which depend on the optical flow estimate on the inference video, are very jittery whereas our predictions are much smoother. In effect, our method distills a given flow network into a more robust estimator of object motion.

(i) Original

(ii) No Targeting

(iii) Magnifying Black Cat

(a) Original

(b) No Targeting

(c) Targeted

ARFlow

RAFT

ARFlow

RAFT

ARFlow

RAFT

Input (

Bookshelf

)

Input (

)

Input (

Drum

)

Another point to note is that the model of Oh et al. , trained using synthetic data with piece-wise linear motions, tends to suffer when the motion is complex. For example in Figure 3, with the moving fur in the cat sequence, or in Figure 4 with the train tracks. Oh et al. also fails when the motion is extreme, as is in the case of the pole sequence in Figure 3 where it fails to magnify an object to its full range of motion. Our model on the other hand generalizes to these methods quite well and is further improved on such out-of-domain videos with test time adaptation.

Targeted magnification. In Figure 5 we show results of targeted magnification. Given a mask, we can easily magnify just the motion in that mask. This is useful when we want to focus on the motion of a specific object, such as in the cat sequence, or when we want to mask out an object whose motion is too large to be magnified, as is the case with the arm in the bookshelf sequence.

ARFlow and RAFT models. We additionally provide qualitative comparisons between our method trained with the unsupervised, and less powerful, ARFlow optical flow model and our method trained with the supervised RAFT model, in Figure 6. As can be seen, despite making slightly less accurate flow estimates, the ARFlow model is fairly comparable in quality to the RAFT model. This shows that our method can enable a fully self-supervised motion magnification model, in which each component is trained with unlabeled data. For all other figures in the paper, we visualize videos generated by our RAFT model.

Quantitative evaluation on real-world videos. We compare our method to baselines on our realworld test set in Table 1 and Figure 7. We magnify frame pairs by magnification factors of α ranging from 1 to 64. Because FLAVR is only trained for α = 10 , we also evaluate this setting. We achieve the best results on almost all evaluation metrics, even with flow models that we did not train for such as PWC-Net and GMFlow, indicating that we can robustly magnify motions.

Quantitative evaluation on synthetic videos. Additionally, we evaluate our method and baselines on the synthetic test sets. The subpixel subset contains frame pairs with 15 levels of purely translational motion varying from 0.04 pixels to 1.0 pixel, and with a fixed target magnified motion magnitude of 10 pixels. The noise subset contains 21 groups of frame pairs with simulated photon noise injected at increasing multiplicative factors, all with a max motion of 2 pixels and fixed magnified motion magnitude of 10 pixels. We report the motion error, magnification error, and SSIM in Table 2 and Figure 8. To compute the motion and magnification error, we use GMFlow [65] trained

2 0

2 1

2 2

2 3

2 4

2 5

2 6

Magnification Factor

2

3

2

1

2 1

2 3

2 5

Motion Error

Oh

et

al

.

Ours (ARFlow)

Ours (RAFT)

(a) PWC-Net

2 0

2 1

2 2

2 3

2 4

2 5

2 6

Magnification Factor

2

3

2

1

2 1

2 3

2 5

Motion Error

(b) RAFT (FlyingThings)

2 0

2 1

2 2

2 3

2 4

2 5

2 6

Magnification Factor

2

3

2

1

2 1

2 3

2 5

Motion Error

(c) GMFlow (FlyingThings)

2 0

2 1

2 2

2 3

2 4

2 5

2 6

Magnification Factor

2

3

2

1

2 1

2 3

2 5

Motion Error

(d) GMFlow (Sintel)

2

5

2

4

2

3

2

2

2

1

2 0

Input Motion Amount (px)

2 1

2 2

2 3

2 4

2 5

Motion Error

Oh

et

al

.

Ours (ARFlow)

Ours (RAFT)

(a) Motion Error for Subpixel Test

2

5

2

4

2

3

2

2

2

1

2 0

Input Motion Amount (px)

2

1

2 0

SSIM

(b) SSIM for Subpixel Test

10

2

10

1

10 0

10 1

10 2

Noise Factor

10 0

10 1

10 2

10 3

Motion Error

(c) Motion Error for Noise Test

10

2

10

1

10 0

10 1

10 2

Noise Factor

10

1

10 0

SSIM

(d) SSIM for Noise Test

with FlyingThings [37]. We achieve satisfactory results on the motion error metrics and lag slightly behind on SSIM.

5 Discussion

We present a method for learning to magnify subtle motions through self-supervised learning. We demonstrated its effectiveness through a variety of experiments, using quantitative evaluations on real and synthetic data, and through qualitative results on videos containing complex motions. We also showed that the flexibility of our model allowed it to be extended through test-time adaptation and targeted motion magnification. We see our work as opening new directions in visualizing subtle motions. First, while we have shown one method for defining the loss function, based on optical flow, our approach could be applied to other differentiable motion estimation techniques, such as emerging

methods for long-range tracking [9, 22, 2]. Second, the flexibility of our model opens the possibility for other 'user in the loop' extensions, beyond allowing for segmentation-based magnification.

Broader impacts. Motion magnification methods have a range of applications, such as amplifying subtle motions in biology and engineering [57], amplifying microexpressions for assistive technology [63], assisting measurement techniques [41], and for video forensics [6, 7]. Please see our website for potential applications. It also has the potential negative use of revealing body motions that a person may think are undetectable, such as one's pulse [64, 1], which may impinge on privacy.

Limitations. The performance of our model is closely tied to the limitations and capabilities of the underlying optical flow estimator. We share the limitation with existing work [40] that our method works by magnifying motion between the first frame and every subsequent frame, which may pose challenges in the presence of occlusion and disocclusion. The magnification loss (Eq. 3) may still be applicable in these cases, since modern optical flow models are trained to track occluded pixels. However, the photoconsistency assumption (Eq. 4) will no longer hold. For simplicity, we do not inpaint missing image regions [32], and we use distance between pixels in lieu of a perceptual loss or a generative model. This simplicity has potential advantages for visualization applications, since it avoids hallucinating scene structure that is not present in the original images, but it also sometimes results in undesirable artifacts.

Acknowledgements. We thank Tae-Hyun Oh for the extensive discussions and data. We also thank Byung-Ki Kwon, Tarun Kalluri, Long Mai, and Stella Yu for helpful discussions. Daniel Geng is supported by a National Science Foundation Graduate Research Fellowship under Grant No. 1841052. This work was supported in part by DARPA Semafor, Program No. HR001120C0123. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.

References

G. Balakrishnan, F. Durand, and J. Guttag. Detecting pulse from head motions in video. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3430-3437, 2013. 10

Z. Bian, A. Jabri, A. A. Efros, and A. Owens. Learning pixel trajectories with multiscale contrastive random walks. CVPR , 2022. 3, 10

T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High accuracy optical flow estimation based on a theory for warping. In European conference on computer vision , pages 25-36. Springer, 2004. 3

M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8748-8757, 2019. 16

J. G. Chen, N. Wadhwa, Y.-J. Cha, F. Durand, W. T. Freeman, and O. Buyukozturk. Modal identification of simple structures with high-speed video using motion magnification. Journal of Sound and Vibration , 345:58-71, 2015. 6

U. A. Ciftci and I. Demir. How do deepfakes move? motion magnification for deepfake source detection. arXiv preprint arXiv:2212.14033 , 2022. 10

V. Conotter, E. Bodnari, G. Boato, and H. Farid. Physiologically-based detection of computer generated faces in video. In 2014 IEEE International Conference on Image Processing (ICIP) , pages 248-252. IEEE, 2014. 10

A. Dave, T. Khurana, P. Tokmakov, C. Schmid, and D. Ramanan. Tao: A large-scale benchmark for tracking any object. In European Conference on Computer Vision , 2020. 6, 16

C. Doersch, A. Gupta, L. Markeeva, A. Recasens, L. Smaira, Y. Aytar, J. Carreira, A. Zisserman, and Y. Yang. Tap-vid: A benchmark for tracking any point in a video. Advances in Neural Information Processing Systems , 2022. 10

A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision , pages 2758-2766, 2015. 1

M. Elgharib, M. Hefeeda, F. Durand, and W. T. Freeman. Video magnification in presence of large motions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4119-4127, 2015. 3

M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision , 111:98-136, 2015. 3, 6

H. Fan, L. Lin, F. Yang, P. Chu, G. Deng, S. Yu, H. Bai, Y. Xu, C. Liao, and H. Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5374-5383, 2019. 16

P. Felzenszwalb and D. Huttenlocher. Efficient belief propagation for early vision. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. , volume 1, pages I-I. IEEE, 2004. 3

B. Y. Feng, H. AlZayer, M. Rubinstein, W. T. Freeman, and J.-B. Huang. Visualizing subtle motions with time-varying neural fields. In International Conference on Computer Vision (ICCV) , 2023. 3

P. Fischer, A. Dosovitskiy, E. Ilg, P. Häusser, C. Hazırba¸ s, V . Golkov, P. Van der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. arXiv preprint arXiv:1504.06852 , 2015. 3

P. Flotho, C. Heiss, G. Steidl, and D. J. Strauss. Lagrangian motion magnification with double sparse optical flow decomposition. arXiv preprint arXiv:2204.07636 , 2022. 2

Y. Gandelsman, Y. Sun, X. Chen, and A. Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems , 35:29374-29385, 2022. 5

S. Gao, Y. Feng, L. Yang, X. Liu, Z. Zhu, D. Doermann, and B. Zhang. Magformer: Hybrid video motion magnification transformer from eulerian and lagrangian perspectives. 2022. 2, 3

D. Geng, M. Hamilton, and A. Owens. Comparing correspondences: Video prediction with correspondencewise losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3365-3376, 2022. 3

A. Goyal, A. Mousavian, C. Paxton, Y.-W. Chao, B. Okorn, J. Deng, and D. Fox. Ifor: Iterative flow minimization for robotic object rearrangement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022. 3

A. W. Harley, Z. Fang, and K. Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision , 2022. 10

B. Horn and B. Schunck. Determining optical flow. In Artificial Intelligence , pages 185-203, 1981. 3, 21

T.-W. Hui, X. Tang, and C. C. Loy. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8981-8989, 2018. 3

E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2462-2470, 2017. 3

A. Jabri, A. Owens, and A. Efros. Space-time correspondence as a contrastive random walk. Advances in neural information processing systems , 33:19545-19560, 2020. 2, 5

R. Jonschkowski, A. Stone, J. T. Barron, A. Gordon, K. Konolige, and A. Angelova. What matters in unsupervised optical flow. arXiv preprint arXiv:2006.04902 , 2020. 1, 3

T. Kalluri, D. Pathak, M. Chandraker, and D. Tran. Flavr: Flow-agnostic video representations for fast frame interpolation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2071-2082, 2023. 3, 7, 15, 18

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643 , 2023. 2, 5

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014. 3, 6

C. Liu et al. Beyond pixels: exploring new representations and applications for motion analysis . PhD thesis, Massachusetts Institute of Technology, 2009. 3

C. Liu, A. Torralba, W. T. Freeman, F. Durand, and E. H. Adelson. Motion magnification. ACMtransactions on graphics (TOG) , 24(3):519-526, 2005. 1, 2, 4, 10

L. Liu, J. Zhang, R. He, Y. Liu, Y. Wang, Y. Tai, D. Luo, C. Wang, J. Li, and F. Huang. Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR) , 2020. 1, 2, 6, 15

P. Liu, I. King, M. R. Lyu, and J. Xu. Ddflow: Learning optical flow with unlabeled data distillation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 8770-8777, 2019. 3

B. D. Lucas, T. Kanade, et al. An iterative image registration technique with an application to stereo vision. Vancouver, British Columbia, 1981. 3

L. Mai and F. Liu. Motion-adjustable neural implicit video representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10738-10747, 2022. 7, 15, 16, 18

N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4040-4048, 2016. 9, 15

M. Menze and A. Geiger. Object scene flow for autonomous vehicles. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3061-3070, 2015. 15

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 2021. 5

T.-H. Oh, R. Jaroensri, C. Kim, M. Elgharib, F. Durand, W. T. Freeman, and W. Matusik. Learning-based video motion magnification. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 633-648, 2018. 1, 2, 3, 4, 5, 6, 7, 9, 10, 15, 18, 19

E. Pérez and E. Zappa. Video motion magnification to improve the accuracy of vision-based vibration measurements. IEEE Transactions on Instrumentation and Measurement , 71:1-12, 2022. 7, 10

J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675 , 2017. 6, 16

A. Ranjan and M. J. Black. Optical flow estimation using a spatial pyramid network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4161-4170, 2017. 3

Z. Ren, J. Yan, B. Ni, B. Liu, X. Yang, and H. Zha. Unsupervised deep learning for optical flow estimation. In Thirty-First AAAI Conference on Artificial Intelligence , 2017. 3

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015. 2, 5, 15

G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 510-526. Springer, 2016. 16

E. P. Simoncelli and W. T. Freeman. The steerable pyramid: A flexible architecture for multi-scale derivative computation. In Proceedings., International Conference on Image Processing , volume 3, pages 444-447. IEEE, 1995. 3

J. Singh, S. Murala, and G. Kosuru. Lightweight network for video motion magnification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2041-2050, 2023. 3

D. Sun, S. Roth, and M. J. Black. Secrets of optical flow estimation and their principles. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 2432-2439. IEEE, 2010. 3, 21

D. Sun, S. Roth, and M. J. Black. A quantitative analysis of current practices in optical flow estimation and the principles behind them. International Journal of Computer Vision , 106:115-137, 2014. 21

D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8934-8943, 2018. 1, 3, 9

Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning , pages 9229-9248. PMLR, 2020. 2, 5

R. Szeliski. Computer vision: algorithms and applications . Springer Nature, 2022. 4

Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 402-419. Springer, 2020. 1, 2, 6, 15

Z. Teed and J. Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems , 34:16558-16569, 2021. 3

B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016. 16

N. Wadhwa, J. G. Chen, J. B. Sellon, D. Wei, M. Rubinstein, R. Ghaffari, D. M. Freeman, O. Büyüköztürk, P. Wang, S. Sun, et al. Motion microscopy for visualizing and quantifying small motions. Proceedings of the National Academy of Sciences , 114(44):11639-11644, 2017. 10

N. Wadhwa, M. Rubinstein, F. Durand, and W. T. Freeman. Phase-based video motion processing. ACM Transactions on Graphics (TOG) , 32(4):1-10, 2013. 1, 3, 4, 6

N. Wadhwa, M. Rubinstein, F. Durand, and W. T. Freeman. Riesz pyramids for fast phase-based video magnification. In 2014 IEEE International Conference on Computational Photography (ICCP) , pages 1-10. IEEE, 2014. 3, 6

W. Wang, M. Feiszli, H. Wang, and D. Tran. Unidentified video objects: A benchmark for dense, openworld segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10776-10785, 2021. 6, 16

Y. Wang, Y. Yang, Z. Yang, L. Zhao, P. Wang, and W. Xu. Occlusion aware unsupervised learning of optical flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4884-4893, 2018. 3

Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, 2004. 6

M. Wei, W. Zheng, Y. Zong, X. Jiang, C. Lu, and J. Liu. A novel micro-expression recognition approach using attention-based magnification-adaptive networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2420-2424. IEEE, 2022. 3, 10

H.-Y. Wu, M. Rubinstein, E. Shih, J. Guttag, F. Durand, and W. Freeman. Eulerian video magnification for revealing subtle changes in the world. ACM transactions on graphics (TOG) , 31(4):1-8, 2012. 1, 2, 3, 4, 6, 10

H. Xu, J. Zhang, J. Cai, H. Rezatofighi, and D. Tao. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8121-8130, 2022. 1, 3, 8, 9

T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision (IJCV) , 127(8):1106-1125, 2019. 6, 16

G. Yang and D. Ramanan. Volumetric correspondence networks for optical flow. In Advances in neural information processing systems , pages 794-805, 2019. 3

L. Yang, Y. Fan, and N. Xu. The 2nd large-scale video object segmentation challenge - video object segmentation track, Oct. 2019. 6, 16

F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2636-2645, 2020. 16

J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4471-4480, 2019. 19, 20

J. J. Yu, A. W. Harley, and K. G. Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In Computer Vision - ECCV 2016 Workshops, Part 3 , 2016. 3

Y. Zhang, S. L. Pintea, and J. C. Van Gemert. Video acceleration magnification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 529-537, 2017. 3

A1 Video Results

We highly encourage the reader to visit our webpage to see qualitative results and comparisons against other methods. Results on the website include:

Videos magnified at various magnification factors, and side-by-side comparisons against Oh et al. [40], FLAVR [28] and NIVR [36].

Targeted magnification results.

Test-time adaptation results.

Failure cases (see Section A8).

Ablations with no regularization (see Section A6).

A2 Training Details

Architecture. We use a U-Net [45] that takes in 2 frames as input (concatenated channel-wise) and a positional encoding for α (described below) and outputs a single frame. Our U-Net is composed of 5 layers of downsampling and 5 layers of upsampling, with skip connections between corresponding layers during downsampling and upsampling. We use a filter size of 64 at the shallowest layer, with the number of filters doubling at each downsampling, ending with a filter size of 1024 for the coarsest layer. Downsampling is performed via max pooling with a kernel size 2 × 2 and stride 2 , and the upsampling is performed using bilinear upsampling. Each layer is composed of a convolution of kernel size 3 × 3 , batch normalization, and a ReLU activation, all applied twice. We apply a sigmoid activation at the very end to transform the final output into the range (0 , 1) . Our model has a total of 17.3M trainable parameters, and the model of Oh et al. has 0.92M parameters. Note that the model of Oh et al. uses a bespoke architecture designed specifically for motion magnification, consisting of an encoder, a decoder, and a manipulator, and is fundamentally different from ours.

To encode magnification factors for input, we transform a single scalar α indicating the desired magnification factor to a 32 dimensional vector. We use sinusoidal positional embeddings with frequencies ranging geometrically from 2 -3 to 2 7 . In order to feed the embeddings into the U-Net we tile it spatially and concatenate it with the two input frames. The resulting input to the U-Net has a channel size of 38, where 6 come from the two input frames and 32 come from the encoding. At training time the spatially tiled positional embeddings are constant-they all represent the same α value. However, at test time it is possible for the embeddings to vary spatially, allowing for various magnification factors in different areas of the video, and enabling targeted magnification.

Training parameters. The parameter λ color is set to 10, while the magnification factors α range from 1 to 16, sampled geometrically. The learning rate is 3 × 10 -4 , and model training is performed using a batch size of 40 with 4 A40 GPUs and an image size of 512 × 512 . In terms of data augmentation, a random area is initially cropped with a scale within the range of (0 . 7 , 1 . 0) . The cropped area is then resized to dimensions of 512 × 512 . Furthermore, the image is subjected to random horizontal or vertical flipping with a probability of 0.5, as well as random rotation within the range of ( -15 ◦ , 15 ◦ ) . Finally, strong color jittering is applied to the frames. These transformations are applied identically to both frames. For the optical flow model, we select ARFlow [33] pretrained with KITTI 2015 [38] and RAFT [54] pretrained on FlyingThings [37]. We use 5 iterations of the RAFT model to compute the flows required by our losses.

Choice of optical flow models. We implemented our algorithm using two distinct optical flow models, namely RAFT [54] and ARFlow [33]. During the training process, we kept the pretrained optical flow model weights fixed, employing them solely to provide motion estimates for our loss computations and gradients for backpropagation to train the U-Net. We use ARFlow, an unsupervised model, to demonstrate that our method works in a truly self-supervised manner, in which no component is trained with labeled data. We then use RAFT, a strong supervised flow network, to train a high quality, weakly-supervised magnification model that serves as a rough upper bound to the performance of a fully self-supervised model. We note that the comparison of quantitative results of both models indicates that our method has better performance with better optical flow quality, hinting that our framework can be benefit from future improvement of optical flow models.

A3 Dataset Details

Dataset overview. To train and evaluate our model, we compile a large dataset of frame pairs by sampling from various existing datasets: Youtube-VOS-2019 [68], DAVIS [42], Vimeo-90k [66], Tracking Any Object (TAO) [8], and Unidentified Video Objects (UVO) [60].The train set is collected by sampling from the train sets of the above datasets, with the exception of DAVIS, where we use the trainval split due to its smaller size. To construct a real-world test set, a total of 50 frame pairs are randomly collected from each dataset's test set, except for the TAO dataset, in which the validation set is used. Because the TAO dataset is comprised of five distinct datasets (ArgoVerse [4], BDD-100K [69], Charades [46], LaSOT [13], and YFCC100M [56]), we sample 50 frame pairs from each of these sub-datasets. In total, 650 frame pairs are acquired across all datasets for evaluation purposes. Basic statistics of the collected dataset is listed in Table A1.

Sampling and filtering. We use consistent filtering and sampling approaches for both the training dataset and the real-world test set. We use two sampling methods, characterized by different temporal strides (1 or 5) indicating the temporal distance between two sampled frames. Specifically, a stride of 5 means that frame i +5 is sampled to form a frame pair alongside frame i . When using a stride of 1, non-overlapping consecutive frame pairs are sampled. For the datasets Youtube-VOS-2019, DAVIS, and Vimeo-90k, a stride of 1 is used. In the case of TAO and UVO, strides of 1 and 5 are utilized to supplement the dataset with additional samples due to the high framerates in these datasets.

After obtaining raw frame pairs from these datasets, a filtering process is conducted to eliminate frame pairs exhibiting significant object or camera motion. We estimate motion using RAFT and set upper bounds for different quantiles of the per-pixel flow magnitude distribution. These include an upper bound of 20 on the 99.9th percentile, an upper bound of 2 on the 80th percentile, and an upper bound of 0.1 on the 0.01st percentile. We find empirically that these thresholds result in frame pairs that have little camera motion and small object motions. To filter out identical or near-identical frame pairs, an additional lower bound is introduced to filter out frame pairs with a mean squared error (MSE) of less than 10 between the two frames.

After sampling and filtering, we obtain a training dataset of 145k frame pairs. Detailed information regarding the number of sampled raw pairs and the number of selected pairs can be found in Table A1 in our paper.

A4 Figure Parameters

We give a comprehensive summary of the magnification factors used and location of y -t slices or closeups for each sequence in Table A2 and Table A3. Additionally, we also provide the corresponding figure index in our paper, and the frame index, indicating which frame we displayed in our figures. Note that some images are cropped or stretched to optimize their presentation within the figures.

A5 Additional Qualitative Results

Comparison with NIVR. Neural implicit video representations (NIVR) [36] is a method that encodes a video into a neural implicit representation. Additionally, the method uses a phase-based positional encoding which allows for control over motion. However, the method does not give precise control over magnification factors as the manipulation is over the latent positional encodings, and is only shown to perform motion magnification up to a factor of about 2 (our work and previous works can achieve factors of up to 200). Furthermore, the code for their method is unreleased. Therefore we compare our method against their results obtained through correspondence with the authors. We show y -t slices in Figure A1. We achieve comparable performance on y -t slices for the flower and guitar

Original

NIVR

Ours

Input (

Guitar1

)

Original

NIVR

Ours

Input (

Flower

)

Original

NIVR

Ours

Input (

Camel

)

Input (

Cats

)

FLAVR

Ours (TTA)

Input (

Train

)

FLAVR

Ours

FLAVR

Ours

Input (

Bookshelf

)

sequences, and our y -t slice for the camel sequence is crisper than NIVR. Comparison of videos is available on our webpage.

Comparison with FLAVR. We also compare against FLAVR [28], a 3-D U-Net proposed for frame interpolation and downstream tasks including motion magnification. Following correspondence with the authors, we replicate their experiments and finetune the public frame interpolation FLA VR checkpoint for motion magnification by training on a subset of the synthetic dataset of Oh et al. [40] with α ≈ 10 . The results of this comparison are displayed in Figure A2. Our magnification has crisper boundary and clearer visual quality. Comparison of videos is available on our webpage.

A6 Additional Experiments

Full evaluation results. In our main paper, due to space limitations we presented selected groups of evaluation results in tables. We present all evaluation results in the following tables for reference,

w/o regularization

w/ regularization

w/ regularization

w/o regularization

Motion Error

↓

Noise Factor

0.01

0.02

0.03

0.04

0.06

0.10

0.16

0.25

0.40

0.63

1.00

1.59

2.51

3.98

6.31

10.00

15.85

25.12

39.81

63.10

100.00

0.65

0.66

0.69

0.66

0.64

0.65

0.71

0.75

0.76

0.76

0.91

1.30

3.49

2.87

2.85

12.43

35.57

82.03

155.00

190.06

221.32

0.61

0.62

0.66

0.62

0.61

0.63

0.68

0.73

0.73

0.74

0.90

1.22

2.21

1.84

2.51

12.85

31.72

78.00

153.10

182.60

226.37

Oh

1.01

1.00

1.05

0.94

0.97

0.96

0.96

0.96

0.96

0.90

0.94

1.07

1.14

1.45

2.18

8.92

17.20

51.44

119.34

163.99

181.26

Ours (ARFlow)

0.82

0.81

0.85

0.82

0.81

0.79

0.80

0.79

0.74

0.70

0.70

0.80

1.01

1.09

1.38

6.44

16.08

46.03

115.04

135.26

167.77

Ours (RAFT)

0.69

0.67

0.71

0.69

0.69

0.68

0.70

0.71

0.68

0.66

0.68

0.79

0.95

1.00

1.15

6.32

14.49

44.04

107.10

128.67

164.36

Magnification Error

↓

Noise Factor

0.01

0.02

0.03

0.04

0.06

0.10

0.16

0.25

0.40

0.63

1.00

1.59

2.51

3.98

6.31

10.00

15.85

25.12

39.81

63.10

100.00

0.16

0.17

0.16

0.16

0.15

0.16

0.18

0.17

0.20

0.20

0.23

0.42

0.93

1.13

0.84

2.22

6.51

11.34

9.65

2.74

1.90

0.15

0.17

0.16

0.15

0.14

0.16

0.17

0.17

0.19

0.19

0.23

0.39

0.58

0.51

0.74

2.21

6.19

9.40

8.47

2.68

1.68

Oh

0.28

0.30

0.29

0.25

0.26

0.26

0.26

0.25

0.26

0.24

0.25

0.30

0.31

0.37

0.54

1.10

1.72

4.00

3.45

1.71

1.70

Ours (ARFlow)

0.20

0.20

0.22

0.20

0.20

0.19

0.20

0.19

0.19

0.18

0.18

0.22

0.24

0.27

0.34

0.71

1.48

1.97

2.67

1.65

1.69

Ours (RAFT)

0.17

0.17

0.18

0.17

0.17

0.17

0.17

0.17

0.17

0.16

0.17

0.21

0.22

0.24

0.31

0.81

1.32

2.28

1.70

1.50

1.49

SSIM

↑

Noise Factor

0.01

0.02

0.03

0.04

0.06

0.10

0.16

0.25

0.40

0.63

1.00

1.59

2.51

3.98

6.31

10.00

15.85

25.12

39.81

63.10

100.00

0.97

0.97

0.97

0.97

0.97

0.97

0.97

0.96

0.95

0.92

0.85

0.75

0.59

0.42

0.29

0.20

0.14

0.14

0.15

0.18

0.18

0.98

0.98

0.97

0.97

0.98

0.98

0.97

0.97

0.96

0.94

0.90

0.83

0.70

0.53

0.39

0.27

0.19

0.17

0.17

0.19

0.19

Oh

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.97

0.96

0.93

0.87

0.78

0.66

0.55

0.45

0.38

0.34

0.31

Ours (ARFlow)

0.97

0.97

0.96

0.96

0.97

0.97

0.97

0.97

0.96

0.96

0.94

0.92

0.86

0.74

0.61

0.46

0.32

0.21

0.15

0.13

0.10

Ours (RAFT)

0.97

0.97

0.97

0.97

0.97

0.97

0.97

0.97

0.96

0.96

0.94

0.91

0.84

0.72

0.59

0.43

0.30

0.20

0.14

0.12

0.10

including results on real-world videos with different optical flow methods in Table A4 and on all subsets of the synthetic data from Oh et al. [40] in Table A5 and Table A6.

No regularization. To demonstrate the significance of the color loss, L color, we ablate it out and show qualitative results for comparison. In Figure A3, we compare our method with and without a color loss with frames from the magnified videos. Notably, we observe that although the magnified motion appears similar to our method with regularization, the color of the results obtained with λ color = 0 significantly deviates from the correct colors and has many artifacts.

A7 Forward Warp with Inpainting

Implementation. We introduced two forward warp techniques, which we refer to as Warp Nearest and Warp Bilinear , as the Lagrangian magnification baselines. To implement these methods, we initially computed optical flow using the same model applied in our experiments (RAFT trained with FlyingThings). Subsequently, we employed forward warping in either a nearest or bilinear manner, applying optical flow values scaled by magnification factors to relocate pixels within the image. Furthermore, we utilized nonparametric inpainting with the built-in functions from OpenCV to address areas left blank due to pixel relocation.

In addition to using nonparametric inpainting, we explored the option of inpainting with DeepFillv2, a powerful learning-based inpainting method proposed by Yu et al. [70]. Two different inpainting methods imply that inpainting is not the bottleneck for Lagrangian magnification without deep learning, thus highlighting the advantage of our method. Our evaluation results indicate comparable performance between forward warp techniques with different inpainting methods. Since the forward warp method employing nonparametric inpainting yielded slightly better qualitative and quantitative results, we used forward warping with nonparametric inpainting in our main paper.

(i) Original Frame

w Inpainting

(v) Warp Bilinear

w Inpainting

(vi) Ours

(ii) Warp Nearest

wo Inpainting

wo Inpainting

Table A7 and Table A8 present a detailed comparison between the forward warp baselines without inpainting and with the two inpainting methods. In our evaluation of real-world videos, we observed a notable decrease in motion error when using inpainting methods in comparison to the warp baselines without inpainting, as expected. On the other hand, during noise tests, especially when using large noise factors, the warp baselines without inpainting exhibited better performance. Notably, we detected an increase in SSIM with noise factor increasing (see Table A6 for more details), for warp baselines with inpainting in noise tests. Our examination revealed that warp baselines produced distorted frames with significant noise factors and struggled to magnify motion when input motion was minimal. Consequently, the evaluation for forward warping baselines may not be able to reflect the actual performance in such cases. Therefore, given the slight advantage of nonparametric inpainting on real-world evaluations, we show the baselines with nonparametric inpainting in visualizations.

Discussion. The two forward warp methods, implemented with advanced techniques as the motion estimator and the inpainter, serve as powerful baselines for Lagrangian techniques, effectively establishing an upper performance limit for straightforward Lagrangian methods. The magnified frames produced through forward warping closely resemble the input frames, thus exhibiting favorable image quality due to their inherent implementation characteristics. Nevertheless, our method consistently outperforms or matches these baselines across various evaluation metrics. Upon qualitative assessment, our approach excels as well, particularly in complex scenarios involving occlusion and

Original

Oh

Ours

Input (

Flower

)

Original

Oh

Ours

Input (

Guitar2

)

Input (

Camera

)

Ours

8

R

disocclusion or challenging flow patterns. We provide one challenging case in Figure A4, where large motion and occlusion were involved. While the results from forward warping methods with/without inpainting suffer from handling the object boundary and background, our method is still capable to provide a satisfactory magnified frame.

A8 Failure Cases

While our method demonstrates favorable performance in most videos, there are cases where it falls short. One particular failure mode occurs when a video contains thin structures such as tree branches and guitar strings. In these cases it appears the flow network has incorrectly estimated the optical flow during training, causing our model to either 'bleed" motion into the stationary background or fail to magnify the motion at all (see flower and guitar in Figure A5). The failure may be attributed to smoothness assumptions of flow estimators [50, 49, 23], and we expect results to improve as optical flow methods get more accurate. Another common failure is incorrectly magnifying background that has zero flow. This happens when small errors from the optical flow model are magnified, as shown in the camera sequence in Figure A5.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed Lagrangian motion magnification using the pre-trained optical flow network. Magnification loss induces that the optical flow of the magnified frame matches with the optical flow of a given frame by $\\alpha$ times; color loss regularizes the color consistency between a given and magnified frames. Test-time adaptation improves the quality of magnification on out-of-domain. Experiments show competitive performance compared to the prior arts in terms of SSIM and the proposed evaluation metrics.\n\nSTRENGTHS:\n1. Simple and effective algorithm to train motion magnification network using the off-the-shelf optical flow network.\n2. Given the off-the-shelf optical flow network, this approach enables the training on large-scale unlabeled videos.\n3. Targeted magnification and test-time adaptation might provide a better user experience.\n4. The proposed method seems to be independent of the architecture of the neural network.\n\nWEAKNESSES:\n1. The term should be used carefully. I am not sure that the proposed method can be named \"self-supervised\" because the off-the-shelf optical flow network, which is used in experiments, is trained by supervised learning. If authors want to use the term \"self-supervised\", the self-supervised optical flow network should be used in the main experiments and the supervised one would be the strong baseline to be compared; It is not sufficient that the self-supervised optical flow network can be used in theory. \n2. Evaluation metric is limited. To justify the underperformance in SSIM, the authors notice this phenomenon in the last sentence in Table 3, \"DeepMag explicitly trains for SSIM\". The proposed algorithm is also optimized by the proposed evaluation metric, Motion Error. For the same reason, I cannot be convinced about the quantitative results. \n3. It is better to include the limitation to use the optical flow network. The optical flow network is inferior to estimate the subtle motion. It is related to the underperformance in the 0.04px subpixel test of Table 3. Thus, the limitation induced by the optical flow network should be investigated as an ablation study because this subtle motion is important in magnification.\n4. I think that this proposed method is more general than supervised or self-supervised learning because this is determined by which optical flow network is used. How about using the synthetic data together?\n5. Is the network architecture different from DeepMag? As for the control experiments, do the number of parameters affect the performance directly? \n6. I think that DeepMag is sufficient as the strong baseline. However, I wonder why Warp Nearest and Bilinear are used, and the more advanced hand-crafted algorithms [A, B] are not used as baselines. It is because evaluation data might contain large motion?\n\n[A] Phase-Based Video Motion Processing (SIGGRAPH 2013)\n\n[B] Riesz Pyramids for Fast Phase-Based Video Magnification (CVPR 2014)\n\nQUESTIONS:\nMy main concerns are the used term and fair comparison. See the weaknesses part.\n\n=====\n\nI update my rate from 4 to 6 because the authors will reflect the discussion below.\n\nLIMITATIONS:\nLimitations and broader impacts are described at the end of the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a self-supervised model to solve the Lagrangian motion magnification problem without needing ground-truth labels. The network takes as input the two input frames and a magnification factor that ranges from 1 to 16, and outputs a generated frame that has magnified motion from the first frame. Off-the-shelf optical flow networks are used in loss computation for self-supervision. Test-time adaption has been explored to enhance the generation quality. Experiments show promising results.\n\nSTRENGTHS:\n1. Good writing; overall clear. The studied problem has received arguably less attention in the research community, but the related work section is detailed and well-structured, which especially helps the readers to catch up.\n2. The method is very simple and easy to understand.\n3. Experiments show promising results.\n4. The authors promised to release full code upon acceptance.\n\nWEAKNESSES:\n1. The targeting application of this task is not clear. Why is this task important? Which data domains or scenarios do we want it to work? If our goal is just to detect small motions, we can develop optical flow estimation methods that work specifically for small motions. Even for existing state-of-the-art optical flow networks, detecting small motions is generally not a big issue, and it should not be hard to find a way to visualize small optical flow. Why do we need to generate a video? Maybe adding some application examples in the introduction and some results on related datasets will help the reader better understand the background and goal of this task.\n2. There are still some confusions on the method. See questions below. \n3. Some minor edits. See additional comments below.\n\nQUESTIONS:\n1. As you included the optical flow network in the whole computation graph with gradient computations, did you freeze its weights to make sure it does not shift away? If so, please state so explicitly in the paper.\n2. Line 185-186: Why is the positional encoding conditioned on the factor $\\alpha$? How do you do it? It does not make sense to me if no explanations are given.\n3. Test-time adaptation: this trick adds in overhead on inference time. How efficient is it?\n4. Did you tackle the occlusion issue? I believe this is an issue both for new frame generation and optical flow estimation. A simple warping using optical flow may be good enough if there are no occlusions at all.\n\nAdditional comments:\n1. Line 26-28:  \"The datasets and learning procedures that are used by these models are designed to be general-purpose, with a particular focus on ensuring that they apply to a variety of motions, objects, and scenes\". Your cited methods are all supervised methods, which are usually trained on large synthetic datasets that could be totally different from real use cases. The generalization ability of these models is still an open question. I think a better idea is to weaken this statement.\n2. Line 32-34: \"And generate a new image pair whose predicted optical flow is $\\alpha$ times as large as that of the input\". Maybe add that the new image pair should share the same reference frame from the input, to avoid confusions.\n3. Line 33-34: add $(\\alpha \\geq 1)$ to be more clear. Your method also makes sense even if $\\alpha < 1$, so it is better to clarify your work cases.\n4. Line 38: \"it\" -> \"our method\".\n5. Line 61: need a citation for \"Eulerian approaches\".\n6. Line 69: \"it uses it\" -> \"we use it\".\n7. Line 115: It is better to make the symbol $x$ bold like $\\mathbf x$ since it is a vector.\n8. Eq 1: Maybe add that this equation assumes no occlusions.\n9. Fig 6 caption: \"subset\" -> \"subsets\".\n10. Repeated references: [21] and [22], [37] and [38].\n\nLIMITATIONS:\nMaybe need to add occlusions as a key limitation", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an optical-flow-based lagrangian motion magnification method, learned through self-supervised learning. The architecture is very simple -- just a U-Net that inputs two temporally consecutive frames and outputs a motion-magnified image. To train the U-Net, the method uses an off-the-shelf optical flow method, estimates the motion between two frames, and considers the motion as real motion of the scene. Then it penalizes the difference between the estimated magnified motion (constant * estimated motion) and the motion between the reference image and motion-magnified image (i.e. output image). The method demonstrates both good quantitative and qualitative results.\n\nSTRENGTHS:\n+ Comprehensive related work\n\n  The paper provides a comprehensive literature survey, which helps understand previous related work and where this paper positions among them.\n\n+ Implementation details\n\n  Sec. 4.1, Sec. 4.2, and supplementary material provide sufficient implementation details so that it's easy to understand the choices of hyper-parameters, training configuration, dataset curations, and training details. Parts of source codes are also included in the supplementary material, which all help reproduce the proposed methods. \n\n+ In-depth evaluation\n\n  Fig. 5, Fig. 6, Table 2, and Table 3 provide in-depth evaluation of the proposed method and related methods on real-world and synthetic videos. Given that there is no public benchmark and the difficulty of evaluation on this topic, the paper tries its best on providing sufficient evaluation.\n\nWEAKNESSES:\n- How to handle occlusion and disocclusion?\n\n  It seems already stated in the limitation section, but I wonder if the method doesn't explicitly handle occlusion and disocclusion. If it doesn't, then can it be a problem? Does U-Net learn to handle them to some extent? When watching the supplementary video, the model doesn't seem to output hallucinated appearance around the disoccluded region, which seems good.\n\n- Worse SSIM in Table 3\n\n  In Table 3, compared to DeepMag, all metrics are better except for SSIM. I wonder why it's the case. What makes the DeepMag's SSIM better than the proposed method?\n\n- Moving background?\n\n  In the supplementary video (1m14s and 2m12s), I am wondering why the background has motion and it's moving? Is it due to that the optical flow method hallucinates motion in the background and it's used during the training? Can this problem be resolved without using the target segmentation mask?\n  By the way, this is another question: what if $L_{mag}$ in Eq. (3) is applied to the target segmentation objects only and penalizes the background motion to be zero? Can it produce better results and prevent the background from moving?\n\n\nThere are some unresolved concerns but the strength outweighs the weaknesses for now. I would like to give Borderline Accept for now, but the rating could change after the discussion phase.\n\n---\nAll concerns are resolved. Thus I am updating my rating to 7. Accept.\n\nQUESTIONS:\n- Usage of the discriminator loss? I wonder if using an adversarial loss from GAN would help more realistic appearance where flow is not accurate, occlusion happens, or artifacts occur.\n- I wonder if the collected data or at least the information on train/test splits will be available if the paper is accepted.\n\nLIMITATIONS:\n- Probably another limitation would be that the method's success depends on the off-the-shelf optical flow and segmentation methods.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses the classical method of Lagrangian to self-supervise the task of motion magnification. Thanks to the proposed self-supervision-based technique, the proposed method can also be adapted during the test time. As shown in Figure.1, the proposed method is simple, where the optical flow vectors of videos before and after magnification are compared.  The optical flow of the motion amplified video is compared to the scaled (by the amplification factor) optical flow of the original video, to derive the magnification loss. To make the output video color consistent, color loss is also used. Videos are provided in the supplementary material for qualitative analysis.\n\nSTRENGTHS:\n1.\tThe method presented in this paper is simple, straightforward, and meaningful. \n2.\tThe experimental evaluations validate the proposed method. Supplementary videos are helpful.\n3.\tThe source code is also provided in the supplementary, which further highlights the simplicity.\n4.\tLimitations of the method are well discussed, and failure cases are shown.\n5.\tThe paper is well-written and easy to follow.\n\nWEAKNESSES:\n1.\tThe proposed method largely depends on the pre-trained optical flow network. \n2.\tGiven the nature of the addressed problem, its evaluation is known to be difficult. This is reflected in the experiments. \n3.\tThe experiments are conducted in a relatively small amount of video frames, and the paper discusses “out of the distribution” and “test time adaptation”. It would be interesting to see how the method generalizes when training on a large number of videos, before proceeding to discuss the rest.\n\nQUESTIONS:\nHow does the proposed method behave with low-quality optical flow? Does the method improve its performance in difficult videos, when trained on a larger amount of data? I can imagine that when trained on a large collection of videos, the method may generalize and hence also perform well in some difficult cases.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on learning a pair-wise motion magnification model in a self-supervised manner. The authors employ recent optical flow models to estimate the flow fields between the original and the motion magnified image pairs. The UNet concatenates a sinusoidal encoded magnification factor with the original images and generates the magnified image.\n\nThe learning process of the UNet is facilitated by a loss function that enforces consistency between the original and magnified flow fields, as well as the consistency between the backward warped images. To demonstrate the effectiveness of the proposed method, the authors curate a large-scale real-world training set. They conduct evaluations both quantitatively on a synthetic dataset and qualitatively on real-world data. The results demonstrate superior performance over previous supervised methods learned on synthetic data.\n\nSTRENGTHS:\n- The paper is well organized and written. Sec.1 introduces the problem effectively and motivates the design of the method. Sec.2 provides a brief but comprehensive review of the previous methods. In addition to the overall organization, sufficient details (including the code) are given for a better understanding of the method, such as the footnote on page 5.\n\n- The method itself is simple and effective:\n\n  - A simple UNet is a compact solution that avoids complicated operations, e.g., explicit optical flow estimation and inpainting.\n\n  - The magnification factor is concatenated with the input image pair after sinusoidal encoding, which enables regionally varying magnification. This is difficult for a single magnification factor as in [33].\n\n  - The self-supervised learning losses enable online adaptation to a specific sequence for better quality.\n\n- The evaluation is comprehensive and achieves significant improvement over previous methods.\n\n- The curated large-scale real-world dataset will encourage further research in this topic.\n\nWEAKNESSES:\nI do not see significant weakness of the paper since it is simple and effective. The only missing piece I come up with is that since the model itself is simple, the author could make some deeper analysis of a learned UNet to understand the underlying mechanism of the model.\n\nQUESTIONS:\n- In lines 83-84, the authors claim that the method's capabilities are more similar to those of Lagrangian methods. However, the internal workings of the model remain unknown. It would be interesting if the authors could conduct some analysis, such as investigating whether some UNet layers implicitly contain motion cues.\n\n- The warping loss compares warped images. What would happen if we additionally, or only, compare two warped images with I_0?\n\n- In the supplementary video, is the magnification factor applied only to the ground in the jumping sequence?\n\nLIMITATIONS:\nThe limitations have been addressed adequately in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}