SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhaoying Pan* Daniel Geng* Andrew Owens
https://dangeng.github.io/FlowMag
This paper presents a simple, self-supervised method for magnifying subtle motions in video: given an input video and a magnification factor, we manipulate the video such that its new optical flow is scaled by the desired amount. To train our model, we propose a loss function that estimates the optical flow of the generated video and penalizes how far if deviates from the given magnification factor. Thus, training involves differentiating through a pretrained optical flow network. Since our model is self-supervised, we can further improve its performance through test-time adaptation, by finetuning it on the input video. It can also be easily extended to magnify the motions of only user-selected objects. Our approach avoids the need for synthetic magnification datasets that have been used to train prior learning-based approaches. Instead, it leverages the existing capabilities of off-the-shelf motion estimators. We demonstrate the effectiveness of our method through evaluations of both visual quality and quantitative metrics on a range of real-world and synthetic videos, and we show our method works for both supervised and unsupervised optical flow methods.
Motion magnification methods [32, 58, 40] increase the size of tiny motions in a video, revealing subtle details that are difficult to discern with the naked eye. However, existing methods come with significant limitations. Early hand-crafted methods generally require small periodic motions [64, 58] or a human in the loop [32]. More recent supervised learning methods [40] require ground-truth training examples, such as videos before and after magnification, which are difficult to obtain without synthetic data. Creating this synthetic data is a challenging problem, since it seemingly requires capturing all of the possible objects and motions that one might ever want to magnify.
In parallel, the field of motion estimation has addressed many closely related challenges. Modern optical flow networks [65, 54, 51, 10] are designed to track objects undergoing complex motions, both large and small. Most of these methods are trained on supervised datasets that capture a wide variety of objects, but parallel work has shown the effectiveness of unsupervised flow estimation [33, 27]. We ask whether we can use motion estimation models to train magnification models, taking advantage of their existing capabilities and reducing the need for special-purpose training data.
We propose a simple motion magnification model whose supervision signal comes from an off-the-shelf motion estimation model. Our method exploits the fact that optical flow networks are differentiable, and thus can be used as part of a loss function. We train a model to take a pair of video frames and a magnification factor  $\alpha$  as input, and to generate a new pair whose predicted optical flow is  $\alpha$  times as large as that of the input. We simultaneously optimize a regularization loss that preserves the visual appearance of each tracked pixel in the generated video. Notably, our model can
(i) Original Frame
(ii) No-TTA Frame
(i) Original Frame
(ii) Left Fork
(iii) Right Fork
(ii) Magnified
(a) Magnification
Figure 1: Magnifying Motions in Video. (a) We show frames along with closeups (Left) or  $y - t$  slices (Right) from original videos and magnified videos. The spatial location of the  $y - t$  slice is visualized by superimposing a colored line on to the still frames, and the location of closeups is shown with a rectangle colored green. (b) We show the frames from the original video, the magnified frame (No-TTA), the magnified frame after test-time adaptation (TTA), and the  $y - t$  slices of the three videos. Test-time adaptation (i.e., finetuning on the input video) improves generation quality as can be seen in the  $y - t$  slices. (c) We magnify the motions within a user-selected object segmentation [29] and show  $y - t$  slices of the magnified videos on different targets. We use red/green colored lines to indicate the locations of  $y - t$  slices for different targets.
(ii) Magnified
(iii) TTA Frame
(b) Test-time Adaptation
(vi) TTA
(v) No Targeting
(c) Targeted Magnification
(vii) Magnifying Right Fork
be trained solely using real unlabeled videos. The optical flow estimator is only used within the loss function during training—we do not require it at test time.
Our model's simplicity and its ability to be trained solely through self-supervision provides it with several advantages over other approaches. We show that we can improve our generated videos through test-time adaptation [52, 26] by finetuning on a given input video. Our formulation can also be easily extended to magnify the motions of a single object, specified via a user-provided mask. In addition, we show results using both off-the-shelf supervised [54] and unsupervised [33] optical flow methods.
Our method is closely related to Lagrangian magnification methods [32, 19], which tracks individual pixels through a video and then amplifies their motion. While this approach is intuitively appealing and was the basis for the earliest magnification methods, a major shortcoming is that it is not clear how to combine the tracking and synthesis together. Previous Lagrangian approaches [32] often rely on hand-crafted pipelines that combine motion estimation, warping, and inpainting steps. By contrast, our method avoids these pitfalls by using off-the-shelf image-to-image translation architectures [45], and uses tracking only within the loss function.
We demonstrate the effectiveness and flexibility of our model in several ways. First, we show through experiments on real and synthetic videos that the optical flow in the generated videos more closely matches the desired motion than videos generated by baseline approaches, such as methods based on warping or supervised learning [40]. Second, we show qualitatively that our model can successfully magnify motions for a variety of videos, containing both small and large motions. Finally, we show that we can improve generation quality using test-time adaptation, and magnify individual objects using user-supplied segmentation maps.
Lagrangian magnification. Motion magnification methods can be broadly categorized as either Lagrangian [32] or Eulerian [64], a classification borrowed from fluid dynamics. Lagrangian methods explicitly track pixels, then generate new frames by forward warping (i.e., splatting) using magnified velocity estimates. Liu et al. [32] originally proposed the motion magnification problem and solved it using a Lagrangian approach. They modeled motion as feature trajectories and (with human-in-the-loop assistance) cluster pixels by similarity in position, color, and motion. Recent methods have extended Lagrangian magnification with more accurate optical flow [17]. While our approach performs explicit tracking and thus is closely related to Lagrangian methods, we decouple generation from tracking: we produce images using off-the-shelf networks and track only within the loss function. This allows us to avoid some of the challenges of the "warp and inpaint" approach, such as handling occlusions and filling holes.
Eulerian magnification. Eulerian methods magnify motions without explicit motion estimation. Instead, they generate a magnified video by amplifying the temporal changes at fixed locations/pixels. Wu et al. [64] decomposed a video into frequency bands and applied temporal filters to extract a signal at a specific bandpass. The extracted signal is then amplified by a magnification factor and added back to the video. Wadhwa et al. [58] proposed a phase-based method, using complex steerable pyramids [47] to decompose the video and separate the phase from the amplitude to amplify the temporally-bandpassed phases. Later work improves efficiency using Riesz pyramids [59] and removes large motion by decomposing a scene into foreground and background layers [11]. Zhang et al. [72] magnified small motions while ignoring large motions by amplifying the motion field acceleration using second-order temporal filters. Concurrent work [15] extended 2D Eulerian methods to 3D motion magnification. Eulerian methods are well-suited to tracking small motions at high spatial frequencies, and may struggle when handling large motions [58], whereas Lagrangian methods' magnification quality is determined by the quality of the optical flow predictions [64]. Since our method's loss function is defined using optical flow, its capabilities are more similar to those of Lagrangian methods.
Learning-based magnification. Several recent works have proposed learning-based motion magnification models. Oh et al. [40] created a synthetic dataset containing image segments extracted from PASCAL VOC [12] as moving foreground and images from COCO [30] as background, and trained a model to regress a ground truth magnified image from two video frames. Inspired by the steerable pyramid used in Eulerian methods [58], they also proposed an architecture that has inductive biases that encourage it to generate crisp images akin to those in Eulerian magnification, by decomposing the image into a shape and texture representation. Other work has extended this supervised magnification approach using 3D CNNs [28] and lightweight architectures [48], and has magnified microexpressions using attention-based models [63]. In contrast to these approaches, our method is self-supervised and learns from unlabeled videos. Recently, Gao et al. [19] use a very similar supervised learning objective and dataset but augment the model of Oh et al. [40] with inductive biases that encourage Lagrangian-like magnification by adding an attention map that is guided by an optical flow field. Instead of making optical flow part of our architecture, as an inductive bias that aids supervised training, we use it to define a self-supervised loss function.
Motion estimation. The field of motion estimation is closely related to motion magnification. Early work solved linearized models after making color constancy assumptions [35] or solved smoothness-regularized models with variational methods [23]. Later work added robust losses and inference strategies [49, 3, 31, 14]. While these methods are well-suited to the subtle motions considered in motion magnification, it is difficult to use them as part of a loss function (as we do in this work), since it is not straightforward to differentiate through during gradient-based learning. Recent methods based on neural networks address this issue, since they are differentiable and highly accurate. A variety of recent methods train neural networks using real or synthetic optical flow data [16, 25, 43, 24, 67, 51, 65] or unlabeled video data [71, 44, 61, 34, 27, 2]. Our use of a pretrained flow model removes the need for special-purpose supervised motion magnification datasets.
Optical flow within a loss function. We take inspiration from recent work that uses differentiable optical flow models to define loss functions that internally perform motion reasoning. Geng et al. [20] used flow to obtain robustness to small positional errors on image generation tasks. Goyal et al. [21] used flow to measure the distance to a goal state for robotic planning. Other work has backpropagated through optical flow as part of a pose estimation model [55]. By contrast, we use flow to compare the motions in two videos.
We describe traditional Lagrangian motion magnification, and our proposed self-supervised Lagrangian magnification model.
3.1 Lagrangian Motion Magnification Overview
Lagrangian motion magnification methods magnify motion by tracking pixels over time, then resynthesizing the video such that the motion of each pixel has increased by a desired amount. More concretely, a point  $\mathbf{x}$  in the initial frame  $I_0$  of a video might be displaced by motion field  $\mathcal{T}(\mathbf{x};t)$  in
(a) Architecture
(b) Losses
Figure 2: Motion Magnification Model. Given a reference frame  $I_0$ , a frame to be magnified  $I_t$ , and a map of per-pixel magnification factors  $\alpha$ , we predict a magnified frame  $\tilde{I}_t$ . We minimize two losses that each use an off-the-shelf optical flow estimator  $\mathcal{F}(\cdot, \cdot)$ . First, we use a magnification loss  $\mathcal{L}_{\mathrm{mag}}$  that encourages the optical flow of the generated video to be  $\alpha$  times as large as that of the input video. And second, we also include a consistency loss  $\mathcal{L}_{\mathrm{color}}$  that measures the visual similarity of corresponding pixels in  $I_t$  and  $\tilde{I}_t$ .
frame  $t$ . Assuming color constancy, we have:
$$
I _ {t} (\mathbf {x} + \mathcal {T} (\mathbf {x}; t)) = I _ {0} (\mathbf {x}), \tag {1}
$$
where  $I_{t}$  is the frame at time  $t$  and  $I(\cdot)$  indicates pixel access. To generate a version of the video magnified by a factor  $\alpha^1$ , one could synthesize a new frame  $\tilde{I}_t$  by increasing the distance traversed by each pixel:
$$
\tilde {I} _ {t} (\mathbf {x} + \alpha \mathcal {T} (\mathbf {x}; t)) \leftarrow I _ {0} (\mathbf {x}). \tag {2}
$$
For example, early work in Lagrangian motion magnification [32] aimed to achieve this by performing forward warping (splatting). However, this leads to holes and aliasing artifacts [53]. Moreover, there is no mechanism for dealing with occlusions: after warping, background pixels may move in front of the foreground. Finally, such a model would not be able to deal with appearance changes without additional constraints (e.g., due to lighting variation).
3.2 Self-Supervised Lagrangian Magnification
We propose a motion magnification model that avoids the limitations of previous Lagrangian methods, and that can be trained solely using unlabeled video (Figure 2). Consider  $\mathcal{F}(I_0, I_t)$ , the motion field obtained by computing optical flow between a reference frame  $I_0$  and another frame  $I_t$ . Our goal is to generate a magnified frame  $\tilde{I}_t$  such that its predicted flow is  $\alpha \mathcal{F}(I_0, I_t)$ , where  $\alpha$  is the magnification factor. To do this, we optimize an objective to minimize the difference between the estimated flow  $\mathcal{F}(I_0, \tilde{I}_t)$  and the desired flow:
$$
\mathcal {L} _ {\text {m a g}} = \left\| \alpha \mathcal {F} \left(I _ {0}, I _ {t}\right) - \mathcal {F} \left(I _ {0}, \tilde {I} _ {t}\right) \right\| _ {1}, \tag {3}
$$
which we term the magnification loss. When  $\mathcal{F}$  is implemented as a neural network, we can optimize this loss using gradient-based learning methods.
Optical flow models are invariant to a variety of photometric changes (e.g., changes in illumination). Thus, optimizing Eq. 3 alone would result in an underconstrained problem. To ensure that the generated frames match the colors in the input video and to also regularize our problem, we use an additional color loss. This loss ensures that corresponding pixels in generated frame  $\tilde{I}_t$  and original frame  $I_t$  are the same color. To enforce this, we put both frame  $I_t$  and  $\tilde{I}_t$  into correspondence with the common reference frame  $I_0$  by backward warping each frame with their respective flows:  $\mathcal{F}(I_0, I_t)$  and  $\mathcal{F}(I_0, \tilde{I}_t)$ . We then measure the distance between the warped frames. This results in a loss:
$$
\mathcal {L} _ {\text {c o l o r}} = \left\| \operatorname {w a r p} \left(I _ {t}, \mathcal {F} \left(I _ {0}, I _ {t}\right)\right) - \operatorname {w a r p} \left(\tilde {I} _ {t}, \mathcal {F} \left(I _ {0}, \tilde {I} _ {t}\right)\right) \right\| _ {1}. \tag {4}
$$
<sup>1</sup> Note that some previous works [64, 58, 40] define the magnification factor in a way that is equivalent to  $\alpha + 1$  (instead of  $\alpha$ ), which is more conducive to their theoretical analysis.
This loss also disincentivizes adversarial examples against the flow network. Our full loss is a weighted sum of these two losses:
$$
\mathcal {L} = \mathcal {L} _ {\text {m a g}} + \lambda_ {\text {c o l o r}} \mathcal {L} _ {\text {c o l o r}}. \tag {5}
$$
One benefit of putting optical flow into the loss function is that our model needs access to the flow network only during training; at inference time the model simply takes in two frames and outputs a magnified frame. Pseudocode for our method for training and inference can be found in Algorithm 1 and Algorithm 2 respectively.
3.3 Image Generation Architecture
One benefit of our formulation is that our proposed loss (Eq. 5) is independent of the image generation architecture, so that "off-the-shelf" image translation architectures can be used to create a magnified image. This is in contrast to other learning-based methods, which incorporate inductive biases within the network to generate images that closely resemble the input (e.g., by altering the structure of an image while preserving the texture [40]). To generate magnified frames, we use a standard U-Net architecture [45] that takes two input frames (the reference frame and the frame to be magnified) concatenated channel-wise. We encode magnification factor  $\alpha$  using a sinusoidal positional embedding [39] and tile it to match the same spatial dimensions as the input frames. We concatenate this embedding channel-wise to the input frames. Please see Section A2 in the appendix for full architectural details.
3.4 Targeted Magnification
Our model gives us the ability to vary the magnification factor spatially within an image. We achieve this by providing different values of  $\alpha$  for each pixel at inference time. This works even when training with spatially constant alpha maps, due to the fully convolutional nature of our network<sup>2</sup>. As a special case, we magnify the motion of a single object by setting the magnification factor to a value of  $\alpha$  within an object segment, and to 1 everywhere else. In practice, we use the recent Segment Anything Model (SAM) [29] to extract a mask for a given object in the reference frame and then dilate the segmentation mask by a small fixed number of pixels. We provide pseudocode for targeted magnification during inference in Algorithm 2.
Algorithm 1 Pseudocode in a PyTorch-like style for training a U-Net for motion magnification.
# Load data of two-frame videos  
for (im0, im1) in datalloader:  
    # Sample alpha and get embedding  
    a = sample_alpha(min_alpha, max_alpha)  
    pe_a = positional_embedding(a)  
    pe_a = spatial_TILE(pe_a)  
# Predict magnified frame with a network  
input = concat([im0, im1, pe_a])  
im1_mag = UNet(input)  
# Estimate the motion  
F_src = optical_flow(im0, im1)  
F_tgt = optical_flow(im0, im1_mag)  
# Warp the second frame  
warp_im1 = warp(im1, F_src)  
warp_im1_mag = warp(im1_mag, F_tgt)  
# Calculate losses  
mag_loss = 11_loss(a * F_src, F_tgt)  
color_loss = 11_loss(warp_im1, warp_im1_mag)  
loss = mag_loss + weight * color_loss  
lossbackward()  
optimizer step()
Algorithm 2 Pseudocode in a PyTorch-like style for inference (targeted magnification).
im0  $=$  video[0]   
magnified(video  $= [\mathrm{im}0]$    
# Load frames from the input video   
for frame in video[1]: #Get masked embedding for input alpha pe_a  $=$  positional_embedding(a) pe_a  $=$  spatial_TILE(pe_a) pe_a  $=$  mask \*pe_a # Predict magnified frame with a network input  $=$  concat([im0,frame,pe_a]) mag_frame  $=$  UNet(input) magnified Video.append(mag_frame)
3.5 Test-Time Adaptation
Since our model is entirely self-supervised, we can finetune it at inference time on the input video [52, 26, 18]. This allows us to adapt to new motions and content. We find that this can significantly improve results, especially for out-of-domain videos. Previous supervised approaches do not have this capability as they require labeled data. To perform test-time adaptation we take the frames in a given inference video, apply minor cropping and color augmentations, and finetune the model with our loss.
2 While the model could be trained with random spatially-varying  $\alpha$  values, we did not see a qualitative improvement when doing this.
We evaluate our model both quantitatively through experiments with real and synthetic data, and qualitatively on real videos. We highly encourage the reader to view our website, since the magnified motions can be challenging to visualize in static images.
We train two variations of our model. One that uses ARFlow [33], and one that uses RAFT [54] as the optical flow network. Since ARFlow is self-supervised, using it results in a fully self-supervised motion magnification method. RAFT on the other hand is a supervised flow model with very good performance, giving us a weakly-supervised magnification method that serves as a rough upper bound to the performance of a fully self-supervised method. More discussion can be found in Section A2 in the appendix. All qualitative results presented in this paper are from the RAFT model, except in Figure 6 in which we compare the ARFlow and RAFT models.
Because motion magnification is multiplicative, we sample random magnification factors  $\alpha$  exponentially, such that  $\log_2(\alpha) \sim U(\log_2(\alpha_{\mathrm{min}}), \log_2(\alpha_{\mathrm{max}}))$ , where  $\alpha_{\mathrm{min}} = 1$  and  $\alpha_{\mathrm{max}} = 16$ . For larger magnification factors that exceed  $\alpha_{\mathrm{max}}$ , we find that recursive application of our model produces high-quality predictions. More implementation details are available in Section A2 in the appendix.
Because our method is self-supervised, we benefit from a large, diverse dataset of videos. To this end we curate a dataset containing 145k unlabeled frame pairs from several existing datasets, including YouTube-VOS-2019 [68], DAVIS [42], Video-90k [66], Tracking Any Object (TAO) [8], and Unidentified Video Objects (UVO) [60]. We remove frame pairs with large motions (e.g., from the camera or objects), since these frames are less likely to be used in magnification applications. We also remove frame pairs that are near-identical by setting a lower bound on the MSE between the two frames. In addition to a training set, we collect a test set consisting 650 frame pairs for evaluation, which we refer to as real-world test set. We provide more details of dataset collection and filtering in Section A3 of the appendix.
For testing, we also use the synthetic test set from Oh et al. [40], which is generated by compositing objects from PASCAL VOC [12] to backgrounds from COCO [30] at varying levels of subpixel motion and noise. We refer to this test set as the synthetic test set. Finally, we qualitatively assess our method and baselines on various real-world videos, which vary greatly in subject matter, motion complexity, and lighting conditions. These videos include de-facto standard benchmarks for motion magnification used in previous works [64, 58, 59, 5, 40], as well as new videos.
4.3 Metrics and Baselines
Metrics. Similar to Oh et al. [40], we use SSIM [62] as an evaluation metric. However, because SSIM requires ground-truth magnified images, we can only use this metric to evaluate models on the synthetic test set. In order to evaluate our method on the real-world test set, we propose another metric for motion magnification that does not require ground truth, which we term motion error. Inspired by the accuracy and robustness of recent optical flow methods, we assume that the flow estimate from an optical flow method is ground truth and calculate an end-point error between the predicted flow and the desired magnified flow.
This metric is identical to  $\mathcal{L}_{\mathrm{mag}}$  if the same optical flow model is used for evaluation as is used during training. In order to ensure a robust metric, we calculate the motion error metric using a wide range of optical flow networks, including PWC-Net, GMFlow, and RAFT, that have been trained on various datasets. We set the number of iterations for RAFT to 20 during evaluation, while keeping it at 5 during training. This serves the dual purpose of allowing us to train more efficiently, but also evaluate more equitably.