SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In addition, we compute the per-pixel ratio between the flow magnitudes of the unmagnified and magnified frames, and calculate their average deviation from the desired magnification factor  $\alpha$ . This metric, which we refer to as magnification error, was not explicitly trained for and serves as another indicator of magnification quality.
(i) Original
(iv) Oh et al.
(v) Ours
(i) Original
(ii) Warp Nearest
(ii) Warp Nearest
(iii) Warp Bilinear
(iii) Warp Bilinear
(vi) Ours (TTA)
(a) Baby
(vi) Ours (TTA)
(b) Cats
(iii) Warp Bilinear
(vi) Ours (TTA)
(c) Pole
Figure 3: Qualitative Comparison. (Top) We visualize the results of various models by plotting  $y - t$  slices through video volumes. (Bottom) We also show the reference frame and the locations of the  $y - t$  slices. Oh et al. (iv) shows noticeable artifacts, indicated by red circles. In (vi) we see that test time adaptation (TTA) can improve our method on out-of-domain inference videos.
Ours
Input (Train)
Input (Boiler)
Input (Camel)
Oh et al.
Figure 4: Image Quality Comparison. We show magnified frames generated from the method of Oh et al. and our method. Because the model of Oh et al. is trained on a synthetic dataset, it may not generalize to the presence of novel motions such as the camel chewing, or novel scenes, such as the train and boiler sequences.
Baselines. We compare primarily against the method of Oh et al. [40], a neural network based approach trained on a synthetic dataset. Additionally, we implement forward warp baselines that use nearest and bilinear sampling to warp pixels according to the amplified flow, along with a nonparametric inpainting method [41], which we term Warp Nearest and Warp Bilinear respectively. We also compare against FLAVR [28], a 3D U-Net trained for frame interpolation and finetuned on the same synthetic dataset as Oh et al., albeit at a constant magnification factor of  $\alpha = 10$ . Finally, we provide qualitative comparisons in Section A5 of the appendix to Neural Implicit Video Representations (NIVR) [36], a method that fits an implicit representation to a video and displays emergent motion manipulation behavior.
4.4 Comparison with the State-of-the-Art
Visual quality. We show qualitative results on real-world videos. In Figure 4 we show still frames and closeups of results from Oh et al. and our method. And in Figure 3 we plot  $y - t$  slices through the video volume, with one dimension being time and the other being spatial<sup>3</sup>.
One qualitative finding is that while the flow network may be noisy on a specific video, our model trained with the same flow network can be much more robust. This can be seen clearly in the baby sequence of Figure 3. The forward warp methods, which depend on the optical flow estimate on the inference video, are very jittery whereas our predictions are much smoother. In effect, our method distills a given flow network into a more robust estimator of object motion.
3For simplicity, we refer both vertical and horizontal slices to  $y - t$  slices.
Figure 5: Targeted Magnification. Our network is capable of targeting motion magnification. This is useful when we want to focus on a specific object, as in the cat sequence above, or when we want to ignore an object that may be challenging to magnify, such as the quickly moving arm in the bookshelf sequence above.
Figure 6: Qualitative Comparison between Models Trained with ARFlow and RAFT. We present  $y - t$  slices from videos magnified by models trained with ARFlow and RAFT. The model (ARFlow) produces magnifications of comparable quality as the model (RAFT), despite being trained on an unsupervised optical flow method.
Another point to note is that the model of Oh et al., trained using synthetic data with piece-wise linear motions, tends to suffer when the motion is complex. For example in Figure 3, with the moving fur in the cat sequence, or in Figure 4 with the train tracks. Oh et al. also fails when the motion is extreme, as is in the case of the pole sequence in Figure 3 where it fails to magnify an object to its full range of motion. Our model on the other hand generalizes to these methods quite well and is further improved on such out-of-domain videos with test time adaptation.
Targeted magnification. In Figure 5 we show results of targeted magnification. Given a mask, we can easily magnify just the motion in that mask. This is useful when we want to focus on the motion of a specific object, such as in the cat sequence, or when we want to mask out an object whose motion is too large to be magnified, as is the case with the arm in the bookshelf sequence.
ARFlow and RAFT models. We additionally provide qualitative comparisons between our method trained with the unsupervised, and less powerful, ARFlow optical flow model and our method trained with the supervised RAFT model, in Figure 6. As can be seen, despite making slightly less accurate flow estimates, the ARFlow model is fairly comparable in quality to the RAFT model. This shows that our method can enable a fully self-supervised motion magnification model, in which each component is trained with unlabeled data. For all other figures in the paper, we visualize videos generated by our RAFT model.
Quantitative evaluation on real-world videos. We compare our method to baselines on our real-world test set in Table 1 and Figure 7. We magnify frame pairs by magnification factors of  $\alpha$  ranging from 1 to 64. Because FLAVR is only trained for  $\alpha = 10$ , we also evaluate this setting. We achieve the best results on almost all evaluation metrics, even with flow models that we did not train for such as PWC-Net and GMFlow, indicating that we can robustly magnify motions.
Quantitative evaluation on synthetic videos. Additionally, we evaluate our method and baselines on the synthetic test sets. The subpixel subset contains frame pairs with 15 levels of purely translational motion varying from 0.04 pixels to 1.0 pixel, and with a fixed target magnified motion magnitude of 10 pixels. The noise subset contains 21 groups of frame pairs with simulated photon noise injected at increasing multiplicative factors, all with a max motion of 2 pixels and fixed magnified motion magnitude of 10 pixels. We report the motion error, magnification error, and SSIM in Table 2 and Figure 8. To compute the motion and magnification error, we use GMFlow [65] trained
(a) PWC-Net
(c) GMFlow (FlyingThings)
(b) RAFT (FlyingThings)
(d) GMFlow (Sintel)
(a) Motion Error for Subpixel Test
(c) Motion Error for Noise Test
Figure 7: Evaluation on real-world test set. We evaluate the methods with  $\alpha$  ranging from 1 to 64 and various flow methods. Error bars show the standard error. On motion error, our method consistently obtains more accurate magnified motions.
Figure 8: Evaluation on synthetic data. We evaluate our model using the synthetic test dataset from Oh et al. [40]. We use two subsets and two metrics (motion error and SSIM). Error bars show the standard error.
(b) SSIM for Subpixel Test
(d) SSIM for Noise Test
Table 1: Quantitative Evaluation Results on Real-world Test Set. We report the motion error of our method and baselines on our collected evaluation set. A smaller motion error represents better magnification quality. For fair comparison, we show results for optical flow methods not used during training, including PWC-Net [51] and GMFlow [65] trained with FlyingThings [37]. These results are a subset of those in Figure 7. We achieve better motion magnification on almost all evaluation metrics.
Method	PWC-Net	RAFT	GMFlow
α=2	α=4	α=10	α=16	α=64	α=2	α=4	α=10	α=16	α=64	α=2	α=4	α=10	α=16	α=64
Warp Nearest	0.59	1.42	4.15	7.24	35.76	0.40	0.83	2.91	5.49	28.87	0.56	1.26	3.84	6.86	33.61
Warp Bilinear	0.57	1.40	4.12	7.28	35.78	0.35	0.78	2.97	5.52	28.70	0.53	1.24	3.79	6.88	33.72
FLAVR	-	-	4.24	-	-	-	-	3.87	-	-	-	-	4.40	-	-
Oh et al.	0.51	1.37	4.29	7.60	39.49	0.47	1.21	3.86	6.95	37.53	0.52	1.40	4.34	7.64	37.31
Ours (ARFlow)	0.38	1.15	3.70	6.58	34.21	0.33	1.03	3.52	6.33	32.89	0.41	1.30	4.15	7.23	35.02
Ours (RAFT)	0.30	0.95	3.32	6.01	32.01	0.26	0.78	2.82	5.20	29.26	0.37	1.19	3.88	6.73	33.49
Table 2: Quantitative Evaluation Results on Synthetic Videos. We report the motion error, magnification error, and SSIM of our method and baselines on two synthetic evaluation sets. Smaller motion error or magnification error, and larger SSIM indicate better quality. We use GMFlow trained with Flying Things to calculate motion error and magnification error. These results are a subset of those plotted in Figure 8.
Method	Motion Error ↓	Magnification Error ↓	SSIM ↑
Subpixel Test	Noise Test	Subpixel Test	Noise Test	Subpixel Test	Noise Test
0.04px	0.2px	1px	0.01x	1x	100x	0.04px	0.2px	1px	0.01x	1x	100x	0.04px	0.2px	1px	0.01x	1x	100x
Warp Nearest	10.60	3.01	2.12	0.65	0.91	221.32	209.10	22.61	3.12	0.16	0.23	1.90	0.77	0.92	0.97	0.97	0.85	0.18
Warp Bilinear	10.61	3.02	2.12	0.61	0.90	226.37	209.35	22.61	3.10	0.15	0.23	1.68	0.78	0.93	0.97	0.98	0.90	0.19
Oh et al.	9.82	2.93	2.26	1.01	0.94	181.26	193.78	22.05	3.15	0.28	0.25	1.70	0.93	0.97	0.98	0.98	0.97	0.31
Ours (ARFlow)	12.29	3.26	2.35	0.82	0.70	167.77	222.16	20.87	3.05	0.20	0.18	1.69	0.75	0.89	0.94	0.97	0.94	0.10
Ours (RAFT)	10.02	2.80	2.05	0.69	0.68	164.36	175.45	17.01	2.48	0.17	0.17	1.49	0.82	0.90	0.94	0.97	0.94	0.10
with FlyingThings [37]. We achieve satisfactory results on the motion error metrics and lag slightly behind on SSIM.
We present a method for learning to magnify subtle motions through self-supervised learning. We demonstrated its effectiveness through a variety of experiments, using quantitative evaluations on real and synthetic data, and through qualitative results on videos containing complex motions. We also showed that the flexibility of our model allowed it to be extended through test-time adaptation and targeted motion magnification. We see our work as opening new directions in visualizing subtle motions. First, while we have shown one method for defining the loss function, based on optical flow, our approach could be applied to other differentiable motion estimation techniques, such as emerging
methods for long-range tracking [9, 22, 2]. Second, the flexibility of our model opens the possibility for other "user in the loop" extensions, beyond allowing for segmentation-based magnification.
Broader impacts. Motion magnification methods have a range of applications, such as amplifying subtle motions in biology and engineering [57], amplifying microexpressions for assistive technology [63], assisting measurement techniques [41], and for video forensics [6, 7]. Please see our website for potential applications. It also has the potential negative use of revealing body motions that a person may think are undetectable, such as one's pulse [64, 1], which may impinge on privacy.
Limitations. The performance of our model is closely tied to the limitations and capabilities of the underlying optical flow estimator. We share the limitation with existing work [40] that our method works by magnifying motion between the first frame and every subsequent frame, which may pose challenges in the presence of occlusion and disocclusion. The magnification loss (Eq. 3) may still be applicable in these cases, since modern optical flow models are trained to track occluded pixels. However, the photoconsistency assumption (Eq. 4) will no longer hold. For simplicity, we do not inpaint missing image regions [32], and we use distance between pixels in lieu of a perceptual loss or a generative model. This simplicity has potential advantages for visualization applications, since it avoids hallucinating scene structure that is not present in the original images, but it also sometimes results in undesirable artifacts.
Acknowledgements. We thank Tae-Hyun Oh for the extensive discussions and data. We also thank Byung-Ki Kwon, Tarun Kalluri, Long Mai, and Stella Yu for helpful discussions. Daniel Geng is supported by a National Science Foundation Graduate Research Fellowship under Grant No. 1841052. This work was supported in part by DARPA Semafor, Program No. HR001120C0123. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.
[1] G. Balakrishnan, F. Durand, and J. Guttag. Detecting pulse from head motions in video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3430-3437, 2013. 10
[2] Z. Bian, A. Jabri, A. A. Efros, and A. Owens. Learning pixel trajectories with multiscale contrastive random walks. CVPR, 2022. 3, 10
[3] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High accuracy optical flow estimation based on a theory for warping. In European conference on computer vision, pages 25-36. Springer, 2004. 3
[4] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8748-8757, 2019. 16
[5] J. G. Chen, N. Wadhwa, Y.-J. Cha, F. Durand, W. T. Freeman, and O. Buyukozturk. Modal identification of simple structures with high-speed video using motion magnification. Journal of Sound and Vibration, 345:58-71, 2015. 6
[6] U. A. Ciftci and I. Demir. How do deepfakes move? motion magnification for deepfake source detection. arXiv preprint arXiv:2212.14033, 2022. 10
[7] V. Conotter, E. Bodnari, G. Boato, and H. Farid. Physiologically-based detection of computer generated faces in video. In 2014 IEEE International Conference on Image Processing (ICIP), pages 248-252. IEEE, 2014. 10
[8] A. Dave, T. Khurana, P. Tokmakov, C. Schmid, and D. Ramanan. Tao: A large-scale benchmark for tracking any object. In European Conference on Computer Vision, 2020. 6, 16
[9] C. Doersch, A. Gupta, L. Markeeva, A. Recasens, L. Smaira, Y. Aytar, J. Carreira, A. Zisserman, and Y. Yang. Tap-vid: A benchmark for tracking any point in a video. Advances in Neural Information Processing Systems, 2022. 10
[10] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758-2766, 2015. 1
[71] J. J. Yu, A. W. Harley, and K. G. Derpanis. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In Computer Vision - ECCV 2016 Workshops, Part 3, 2016. 3
[72] Y. Zhang, S. L. Pintea, and J. C. Van Gemert. Video acceleration magnification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 529-537, 2017. 3
A1 Video Results
We highly encourage the reader to visit our webpage to see qualitative results and comparisons against other methods. Results on the website include:
- Videos magnified at various magnification factors, and side-by-side comparisons against Oh et al. [40], FLAVR [28] and NIVR [36].
- Targeted magnification results.
- Test-time adaptation results.
- Failure cases (see Section A8).
- Ablations with no regularization (see Section A6).
A2 Training Details
Architecture. We use a U-Net [45] that takes in 2 frames as input (concatenated channel-wise) and a positional encoding for  $\alpha$  (described below) and outputs a single frame. Our U-Net is composed of 5 layers of downsampling and 5 layers of upsampling, with skip connections between corresponding layers during downsampling and upsampling. We use a filter size of 64 at the shallowest layer, with the number of filters doubling at each downsampling, ending with a filter size of 1024 for the coarsest layer. Downsampling is performed via max pooling with a kernel size  $2 \times 2$  and stride 2, and the upsampling is performed using bilinear upsampling. Each layer is composed of a convolution of kernel size  $3 \times 3$ , batch normalization, and a ReLU activation, all applied twice. We apply a sigmoid activation at the very end to transform the final output into the range  $(0,1)$ . Our model has a total of 17.3M trainable parameters, and the model of Oh et al. has 0.92M parameters. Note that the model of Oh et al. uses a bespoke architecture designed specifically for motion magnification, consisting of an encoder, a decoder, and a manipulator, and is fundamentally different from ours.
To encode magnification factors for input, we transform a single scalar  $\alpha$  indicating the desired magnification factor to a 32 dimensional vector. We use sinusoidal positional embeddings with frequencies ranging geometrically from  $2^{-3}$  to  $2^{7}$ . In order to feed the embeddings into the U-Net we tile it spatially and concatenate it with the two input frames. The resulting input to the U-Net has a channel size of 38, where 6 come from the two input frames and 32 come from the encoding. At training time the spatially tiled positional embeddings are constant—they all represent the same  $\alpha$  value. However, at test time it is possible for the embeddings to vary spatially, allowing for various magnification factors in different areas of the video, and enabling targeted magnification.
Training parameters. The parameter  $\lambda_{\mathrm{color}}$  is set to 10, while the magnification factors  $\alpha$  range from 1 to 16, sampled geometrically. The learning rate is  $3\times 10^{-4}$ , and model training is performed using a batch size of 40 with 4 A40 GPUs and an image size of  $512\times 512$ . In terms of data augmentation, a random area is initially cropped with a scale within the range of (0.7, 1.0). The cropped area is then resized to dimensions of  $512\times 512$ . Furthermore, the image is subjected to random horizontal or vertical flipping with a probability of 0.5, as well as random rotation within the range of  $(-15^{\circ},15^{\circ})$ . Finally, strong color jittering is applied to the frames. These transformations are applied identically to both frames. For the optical flow model, we select ARFlow [33] pretrained with KITTI 2015 [38] and RAFT [54] pretrained on FlyingThings [37]. We use 5 iterations of the RAFT model to compute the flows required by our losses.
Choice of optical flow models. We implemented our algorithm using two distinct optical flow models, namely RAFT [54] and ARFlow [33]. During the training process, we kept the pretrained optical flow model weights fixed, employing them solely to provide motion estimates for our loss computations and gradients for backpropagation to train the U-Net. We use ARFlow, an unsupervised model, to demonstrate that our method works in a truly self-supervised manner, in which no component is trained with labeled data. We then use RAFT, a strong supervised flow network, to train a high quality, weakly-supervised magnification model that serves as a rough upper bound to the performance of a fully self-supervised model. We note that the comparison of quantitative results of both models indicates that our method has better performance with better optical flow quality, hinting that our framework can be benefit from future improvement of optical flow models.
Table A1: Dataset Statistics (Train Set). We report basic statistics of our training dataset, including the temporal sampling stride and the number of frame pairs before and after filtering.
YouTube-VOS	DAVIS	Vimeo-90k	TAO	UVO	Total
Sampling Stride	1	1	1	1	5	1	5	-
# Raw Pairs	46,858	3,089	193,836	176,092	175,434	71,836	70,830	737,975
# Selected Pairs	4,050	370	28,734	50,441	29,795	22,462	9,749	145,601
A3 Dataset Details
Dataset overview. To train and evaluate our model, we compile a large dataset of frame pairs by sampling from various existing datasets: Youtube-VOS-2019 [68], DAVIS [42], Vimeo-90k [66], Tracking Any Object (TAO) [8], and Unidentified Video Objects (UVO) [60]. The train set is collected by sampling from the train sets of the above datasets, with the exception of DAVIS, where we use the trainval split due to its smaller size. To construct a real-world test set, a total of 50 frame pairs are randomly collected from each dataset's test set, except for the TAO dataset, in which the validation set is used. Because the TAO dataset is comprised of five distinct datasets (ArgoVerse [4], BDD-100K [69], Charades [46], LaSOT [13], and YFCC100M [56]), we sample 50 frame pairs from each of these sub-datasets. In total, 650 frame pairs are acquired across all datasets for evaluation purposes. Basic statistics of the collected dataset is listed in Table A1.
Sampling and filtering. We use consistent filtering and sampling approaches for both the training dataset and the real-world test set. We use two sampling methods, characterized by different temporal strides (1 or 5) indicating the temporal distance between two sampled frames. Specifically, a stride of 5 means that frame  $i + 5$  is sampled to form a frame pair alongside frame  $i$ . When using a stride of 1, non-overlapping consecutive frame pairs are sampled. For the datasets Youtube-VOS-2019, DAVIS, and Vimeo-90k, a stride of 1 is used. In the case of TAO and UVO, strides of 1 and 5 are utilized to supplement the dataset with additional samples due to the high framereates in these datasets.
After obtaining raw frame pairs from these datasets, a filtering process is conducted to eliminate frame pairs exhibiting significant object or camera motion. We estimate motion using RAFT and set upper bounds for different quantiles of the per-pixel flow magnitude distribution. These include an upper bound of 20 on the 99.9th percentile, an upper bound of 2 on the 80th percentile, and an upper bound of 0.1 on the 0.01st percentile. We find empirically that these thresholds result in frame pairs that have little camera motion and small object motions. To filter out identical or near-identical frame pairs, an additional lower bound is introduced to filter out frame pairs with a mean squared error (MSE) of less than 10 between the two frames.
After sampling and filtering, we obtain a training dataset of 145k frame pairs. Detailed information regarding the number of sampled raw pairs and the number of selected pairs can be found in Table A1 in our paper.
A4 Figure Parameters
We give a comprehensive summary of the magnification factors used and location of  $y - t$  slices or closeups for each sequence in Table A2 and Table A3. Additionally, we also provide the corresponding figure index in our paper, and the frame index, indicating which frame we displayed in our figures. Note that some images are cropped or stretched to optimize their presentation within the figures.
A5 Additional Qualitative Results
Comparison with NIVR. Neural implicit video representations (NIVR) [36] is a method that encodes a video into a neural implicit representation. Additionally, the method uses a phase-based positional encoding which allows for control over motion. However, the method does not give precise control over magnification factors as the manipulation is over the latent positional encodings, and is only shown to perform motion magnification up to a factor of about 2 (our work and previous works can achieve factors of up to 200). Furthermore, the code for their method is unreleased. Therefore we compare our method against their results obtained through correspondence with the authors. We show  $y-t$  slices in Figure A1. We achieve comparable performance on  $y-t$  slices for the flower and guitar
Table A2: Information on  $y - t$  Slices of Magnified Sequences. We represent the location of  $y - t$  slices with the upper left  $\left( {{x}_{1},{y}_{1}}\right)$  coordinate and bottom right coordinate  $\left( {{x}_{2},{y}_{2}}\right)$  for every sequence appeared in our paper. We also provide magnification factors for each  $y - t$  slices. For frames with two  $y - t$  slices in the same frame, we report the left and right locations accordingly, and mark with "L" for left, "R" for right in the table. For sequences magnified by different magnification factors, we report information for every motion magnification factor.