SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sequence	Drum	Pole	Tuning Forks (L)	Tuning Forks (R)	Baby	Cats (L)
Figure	Figure 1,6	Figure 1,3	Figure 1,6, A3	Figure 1,6, A3	Figure 3	Figure 5
α	5	20	5	5	20	20
Upper Left	(180, 250)	(20, 120)	(175, 200)	(340, 260)	(510, 212)	(280, 160)
Bottom Right	(181, 310)	(140, 121)	(315, 201)	(480, 261)	(511, 292)	(281, 300)
Sequence	Cats (R)	Bookshelf (L)	Bookshelf (R)	Flower	Guitar1	Camel
Figure	Figure 3,5	Figure 5,6	Figure 5	Figure A1	Figure A1	Figure A1
α	20	15	15	2	2	2
Upper Left	(710, 130)	(240, 340)	(560, 260)	(260, 50)	(20, 120)	(250, 90)
Bottom Right	(711, 270)	(241, 400)	(561, 320)	(350, 51)	(21, 160)	(280, 91)
Sequence	Cats (R)	Bookshelf (L)	Boiler	Flower	Guitar2
Figure	Figure A2	Figure A2	Figure A3,A4	Figure A5	Figure A5
α	10	10	30	20	25
Upper Left	(710, 130)	(240, 340)	(100, 200)	(240, 10)	(280, 50)
Bottom Right	(711, 270)	(241, 400)	(200, 201)	(241, 50)	(281, 100)
Table A3: Information on Closeups of Magnified Sequences. We represent the location of closeups with the upper left  $(x_{1},y_{1})$  coordinate and bottom right coordinate  $(x_{2},y_{2})$ . We include frame index that appears in our figures, and frame index starts from 1 for a video. We also provide magnification factors for each closeups in our paper.
Sequence	Train	Pole	Train	Camel	Boiler
Figure	Figure 1	Figure 1	Figure 4	Figure 4	Figure 4
α	20	20	20	20	30
Upper Left	(20, 520)	(0, 0)	(50, 520)	(280, 50)	(100, 120)
Bottom Right	(300, 710)	(200, 200)	(240, 710)	(420, 300)	(350, 576)
Frame Index	417	1, 190	417	30	19
Sequence	Train	Tuning Forks	Boiler	Boiler	Camera
Figure	Figure A2	Figure A3	Figure A3	Figure A4	Figure A5
α	10	5	30	30	75
Upper Left	(50, 520)	-	-	(80, 120)	(0, 0)
Bottom Right	(240, 710)	-	-	(280, 320)	(45, 200)
Frame Index	417	10	19	19	16
Table A4: Quantitative Evaluation Results on Real-world Videos. We report full evaluation results on real-world videos. Performance is measured by motion error and magnification error for various optical flow estimators.
Motion Error ↓	Magnification Error ↓
α=1	α=2	α=4	α=8	α=16	α=32	α=64	α=1	α=2	α=4	α=8	α=16	α=32	α=64
PWC-Net	Warp Nearest	0.32	0.59	1.42	3.16	7.24	16.24	35.76	0.56	1.07	2.28	4.65	9.54	19.94	43.51
Warp Bilinear	0.28	0.57	1.40	3.16	7.28	16.18	35.78	0.63	1.08	2.21	4.51	9.45	19.82	43.37
Oh et al.	0.05	0.51	1.37	3.27	7.60	17.52	39.49	0.20	0.80	2.06	4.51	9.65	21.58	50.36
Ours (ARFlow)	0.04	0.38	1.15	2.81	6.58	15.18	34.21	0.13	0.74	2.13	4.52	9.19	19.68	41.51
Ours (RAFT)	0.05	0.30	0.95	2.48	6.01	14.05	32.01	0.13	0.55	1.56	3.62	7.74	16.22	34.82
RAFT	Warp Nearest	0.24	0.40	0.83	2.19	5.49	12.71	28.87	0.65	1.18	2.25	4.33	8.79	18.04	39.63
Warp Bilinear	0.18	0.35	0.78	2.20	5.52	12.81	28.70	0.76	1.15	2.08	4.46	9.01	17.90	38.22
Oh et al.	0.05	0.48	1.21	2.92	6.94	16.39	37.55	0.29	0.83	1.90	4.20	9.33	21.44	51.07
Ours (ARFlow)	0.04	0.33	1.03	2.64	6.33	14.63	32.89	0.19	0.75	2.06	4.68	10.25	22.51	47.28
Ours (RAFT)	0.04	0.26	0.78	2.10	5.20	12.66	29.26	0.16	0.56	1.58	3.79	8.34	17.49	35.59
GMFlow (FlyingThings)	Warp Nearest	0.27	0.56	1.26	2.93	6.86	15.43	33.61	0.61	1.20	2.52	5.14	10.43	21.17	44.56
Warp Bilinear	0.24	0.53	1.24	2.94	6.88	15.46	33.72	0.64	1.17	2.42	5.05	10.38	21.44	45.14
Oh et al.	0.09	0.52	1.40	3.31	7.64	16.84	37.31	0.47	0.92	2.11	4.75	10.53	22.74	51.99
Ours (ARFlow)	0.07	0.41	1.30	3.18	7.23	16.68	35.02	0.28	0.82	2.18	4.64	9.72	21.47	44.74
Ours (RAFT)	0.07	0.37	1.19	2.98	6.73	15.32	33.49	0.26	0.65	1.81	4.36	9.11	19.62	42.17
GMFlow (Sintel)	Warp Nearest	0.26	0.55	1.43	3.64	8.02	17.32	36.69	0.66	1.36	3.25	7.73	16.76	33.61	66.49
Warp Bilinear	0.23	0.51	1.39	3.45	8.04	17.19	36.84	0.96	1.70	3.57	7.79	16.64	33.53	66.57
Oh et al.	0.06	0.53	1.49	3.60	8.13	17.53	37.31	0.65	1.09	2.40	5.26	11.28	24.01	51.71
Ours (ARFlow)	0.05	0.39	1.38	3.49	7.87	17.25	36.10	0.26	0.96	2.91	6.45	13.27	28.34	55.73
Ours (RAFT)	0.06	0.39	1.24	3.22	7.43	16.33	34.44	0.28	0.72	2.03	4.94	10.88	23.45	47.67
Input (Flower)
Input (Guitar1)
Figure A1: Qualitative Comparison with NIVR. NIVR [36] magnifies a learned implicit representation, which does not map cleanly to a precise magnification factor. We found using  $\alpha = 2$  with our method gave comparable magnification to the results from NIVR. Because there is no publicly released code for NIVR, we compare against videos and results from the NIVR website. Above we visualize the  $y - t$  slices of the original videos, NIVR videos, and our videos for comparison.
Input (Cats)
Ours (TTA)
Figure A2: Qualitative Comparison with FLAVR. We use  $\alpha = 10$  for our method and compare the results with the magnified videos of FLAVR. We show the  $y - t$  slices of FLAVR videos and our videos. We also show a closeup in the train sequence, shown in the green rectangle.
Input (Bookshelf)
Table A5: Quantitative Evaluation Results on Synthetic Videos. We report the evaluation results on the sub-pixel test with synthetic videos with all different input motion amount, with three metrics including motion error, magnification error, and SSIM.
Motion Error ↓
Input Motion	0.04px	0.05px	0.06px	0.08px	0.10px	0.13px	0.16px	0.20px	0.25px	0.32px	0.40px	0.50px	0.63px	0.79px	1.00px
Warp Nearest	10.60	8.34	6.80	5.45	4.77	4.13	3.80	3.01	2.80	2.56	2.57	2.49	2.21	2.24	2.12
Warp Bilinear	10.61	8.34	6.80	5.44	4.77	4.13	3.58	3.02	2.80	2.56	2.57	2.49	2.20	2.23	2.12
Oh et al.	9.82	7.93	6.46	5.17	4.44	3.85	3.60	2.93	2.87	2.51	2.57	2.61	2.44	2.26	2.26
Ours (ARFlow)	12.29	9.86	8.16	6.38	5.37	4.48	3.99	3.26	3.02	2.65	2.69	2.53	2.42	2.42	2.35
Ours (RAFT)	10.02	7.96	6.55	5.22	4.50	3.86	3.55	2.80	2.68	2.39	2.38	2.29	2.16	2.15	2.05
Magnification Error ↓
Input Motion	0.04px	0.05px	0.06px	0.08px	0.10px	0.13px	0.16px	0.20px	0.25px	0.32px	0.40px	0.50px	0.63px	1.00px
Warp Nearest	209.10	155.27	117.06	85.36	62.23	45.11	32.52	22.61	16.61	12.75	10.71	7.98	5.29	4.59	3.12
Warp Bilinear	209.35	153.32	116.98	85.40	62.24	45.37	30.09	22.61	16.55	12.73	10.70	7.99	5.29	4.58	3.10
Oh et al.	193.78	145.86	106.22	79.79	56.42	42.23	30.96	22.05	17.37	12.50	10.53	7.90	5.55	4.39	3.15
Ours (ARFlow)	222.16	165.24	122.15	86.58	62.42	42.24	29.68	20.87	15.86	11.45	8.78	6.67	5.03	3.99	3.05
Ours (RAFT)	175.45	133.33	97.06	69.86	52.10	36.56	25.42	17.01	13.36	9.86	8.23	5.94	4.20	3.32	2.48
SSIM ↑
Input Motion	0.04px	0.05px	0.06px	0.08px	0.10px	0.13px	0.16px	0.20px	0.25px	0.32px	0.40px	0.50px	0.63px	0.00px
Warp Nearest	0.77	0.80	0.82	0.86	0.89	0.90	0.91	0.92	0.93	0.94	0.95	0.95	0.96	0.96	0.97
Warp Bilinear	0.78	0.81	0.83	0.86	0.89	0.91	0.91	0.93	0.94	0.94	0.95	0.95	0.96	0.97	0.97
Oh et al.	0.93	0.94	0.95	0.96	0.97	0.97	0.97	0.97	0.97	0.97	0.98	0.98	0.98	0.98	0.98
Ours (ARFlow)	0.75	0.77	0.79	0.82	0.84	0.86	0.87	0.89	0.89	0.90	0.92	0.92	0.93	0.93	0.94
Ours (RAFT)	0.82	0.83	0.84	0.86	0.88	0.89	0.89	0.90	0.89	0.90	0.91	0.91	0.92	0.93	0.94
sequences, and our  $y$ - $t$  slice for the camel sequence is crisper than NIVR. Comparison of videos is available on our webpage.
Comparison with FLAVR. We also compare against FLAVR [28], a 3-D U-Net proposed for frame interpolation and downstream tasks including motion magnification. Following correspondence with the authors, we replicate their experiments and finetune the public frame interpolation FLAVR checkpoint for motion magnification by training on a subset of the synthetic dataset of Oh et al. [40] with  $\alpha \approx 10$ . The results of this comparison are displayed in Figure A2. Our magnification has crisper boundary and clearer visual quality. Comparison of videos is available on our webpage.
A6 Additional Experiments
Full evaluation results. In our main paper, due to space limitations we presented selected groups of evaluation results in tables. We present all evaluation results in the following tables for reference,
w/o regularization
w/ regularization
Figure A3: Qualitative Comparison with No Regularization Experiments. We show magnified frames and  $y - t$  slices from our model with and without the color loss regularization.
w/ regularization
Table A6: Quantitative Evaluation Results on Synthetic Videos. We report the evaluation results on the noise test with synthetic videos with all different noise factors, with three metrics including motion error, magnification error, and SSIM.
Motion Error ↓
Noise Factor	0.01	0.02	0.03	0.04	0.06	0.10	0.16	0.25	0.40	0.63	1.00	1.59	2.51	3.98	6.31	10.00	15.85	25.12	39.81	63.10	100.00
Warp Nearest	0.65	0.66	0.69	0.66	0.64	0.65	0.71	0.75	0.76	0.76	0.91	1.30	3.49	2.87	2.85	12.43	35.57	82.03	155.00	190.06	221.32
Warp Bilinear	0.61	0.62	0.66	0.62	0.61	0.63	0.68	0.73	0.73	0.74	0.90	1.22	2.21	1.84	2.51	12.85	31.72	78.00	153.10	182.60	226.37
Oh et al.	1.01	1.00	1.05	0.94	0.97	0.96	0.96	0.96	0.96	0.90	0.94	1.07	1.14	1.45	2.18	8.92	17.20	51.44	119.34	163.99	181.26
Ours (ARFlow)	0.82	0.81	0.85	0.82	0.81	0.79	0.80	0.79	0.74	0.70	0.70	0.80	1.01	1.09	1.38	6.44	16.08	46.03	115.04	135.26	167.77
Ours (RAFT)	0.69	0.67	0.71	0.69	0.69	0.68	0.70	0.71	0.68	0.66	0.68	0.79	0.95	1.00	1.15	6.32	14.49	44.04	107.10	128.67	164.36
Magnification Error ↓
Noise Factor	0.01	0.02	0.03	0.04	0.06	0.10	0.16	0.25	0.40	0.63	1.00	1.59	2.51	3.98	6.31
Warp Nearest	0.16	0.17	0.16	0.16	0.15	0.16	0.18	0.17	0.20	0.20	0.23	0.42	0.93	1.13	0.84	2.22	6.51	11.34	9.65	2.74	1.90
Warp Bilinear	0.15	0.17	0.16	0.15	0.14	0.16	0.17	0.17	0.19	0.19	0.23	0.39	0.58	0.51	0.74	2.21	6.19	9.40	8.47	2.68	1.68
Oh et al.	0.28	0.30	0.29	0.25	0.26	0.26	0.26	0.25	0.26	0.24	0.25	0.30	0.31	0.37	0.54	1.10	1.72	4.00	3.45	1.71	1.70
Ours (ARFlow)	0.20	0.20	0.22	0.20	0.20	0.19	0.20	0.19	0.19	0.18	0.18	0.22	0.24	0.27	0.34	0.71	1.48	1.97	2.67	1.65	1.69
Ours (RAFT)	0.17	0.17	0.18	0.17	0.17	0.17	0.17	0.17	0.17	0.16	0.17	0.21	0.22	0.24	0.31	0.81	1.32	2.28	1.70	1.50	1.49
SSIM ↑
Noise Factor	0.01	0.02	0.03	0.04	0.06	0.10	0.16	0.25	0.40	0.63	1.00	1.59	2.51	3.98	6.3	10.00
Warp Nearest	0.97	0.97	0.97	0.97	0.97	0.97	0.97	0.96	0.95	0.92	0.85	0.75	0.59	0.42	0.29	0.20	0.14	0.14	0.15	0.18	0.18
Warp Bilinear	0.98	0.98	0.97	0.97	0.98	0.98	0.97	0.97	0.96	0.94	0.90	0.83	0.70	0.53	0.39	0.27	0.19	0.17	0.17	0.19	0.19
Oh et al.	0.98	0.98	0.98	0.98	0.98	0.98	0.98	0.98	0.98	0.98	0.97	0.96	0.93	0.87	0.78	0.66	0.55	0.45	0.38	0.34	0.31
Ours (ARFlow)	0.97	0.97	0.96	0.96	0.97	0.97	0.97	0.97	0.96	0.96	0.94	0.92	0.86	0.74	0.61	0.46	0.32	0.21	0.15	0.13	0.10
Ours (RAFT)	0.97	0.97	0.97	0.97	0.97	0.97	0.97	0.97	0.96	0.96	0.94	0.91	0.84	0.72	0.59	0.43	0.30	0.20	0.14	0.12	0.10
including results on real-world videos with different optical flow methods in Table A4 and on all subsets of the synthetic data from Oh et al. [40] in Table A5 and Table A6.
No regularization. To demonstrate the significance of the color loss,  $\mathcal{L}_{\mathrm{color}}$ , we ablate it out and show qualitative results for comparison. In Figure A3, we compare our method with and without a color loss with frames from the magnified videos. Notably, we observe that although the magnified motion appears similar to our method with regularization, the color of the results obtained with  $\lambda_{\mathrm{color}} = 0$  significantly deviates from the correct colors and has many artifacts.
A7 Forward Warp with Inpainting
Implementation. We introduced two forward warp techniques, which we refer to as Warp Nearest and Warp Bilinear, as the Lagrangian magnification baselines. To implement these methods, we initially computed optical flow using the same model applied in our experiments (RAFT trained with FlyingThings). Subsequently, we employed forward warping in either a nearest or bilinear manner, applying optical flow values scaled by magnification factors to relocate pixels within the image. Furthermore, we utilized nonparametric inpainting with the built-in functions from OpenCV to address areas left blank due to pixel relocation.
In addition to using nonparametric inpainting, we explored the option of inpainting with DeepFillv2, a powerful learning-based inpainting method proposed by Yu et al. [70]. Two different inpainting methods imply that inpainting is not the bottleneck for Lagrangian magnification without deep learning, thus highlighting the advantage of our method. Our evaluation results indicate comparable performance between forward warp techniques with different inpainting methods. Since the forward warp method employing nonparametric inpainting yielded slightly better qualitative and quantitative results, we used forward warping with nonparametric inpainting in our main paper.
(ii) Warp Nearest wo Inpainting
(iii) Warp Bilinear wo Inpainting
Figure A4: Failure Cases of Forward Warp Baselines. We show one frame from the boiler sequence, along with the magnified frame from warp nearest/bilinear without inpainting, warp nearest/bilinear with nonparametric inpainting. The closeups and  $y - t$  slices obtained through forward warp methods with and without inpainting exhibit difficulties in managing object boundaries and the background. In contrast, our method successfully produces a satisfactory magnified frame under these challenging conditions.
(iv) Warp Nearest w Inpainting
(v) Warp Bilinear w Inpainting
(vi) Ours
Table A7: Evaluation on Real-world Videos for Different Inpainting Methods for Forward Warp. Similar to Table 1, we evaluated and compared among different inpainting methods for both warp nearest and warp bilinear, including no inpainting (labeled as "None"), nonparametric inpainting, and DeepFillv2 [70] inpainting.
Method	Inpainting	PWC-Net	RAFT	GMFlow
α=2	α=4	α=10	α=16	α=64	α=2	α=4	α=10	α=16	α=64	α=2	α=4	α=10	α=16	α=64
Warp Nearest	None	0.92	1.86	5.43	9.12	42.23	0.50	1.18	3.87	7.13	39.82	0.91	1.70	5.29	9.05	50.38
Nonparametric	0.59	1.42	4.15	7.24	35.76	0.40	0.83	2.91	5.49	28.87	0.56	1.26	3.84	6.86	33.61
DeepFillv2	0.61	1.40	4.21	7.44	36.20	0.39	0.84	2.96	5.47	29.21	0.55	1.27	3.91	6.95	33.63
Warp Bilinear	None	0.68	1.49	4.32	7.44	36.93	0.39	0.91	3.07	5.79	30.60	0.66	1.62	4.26	7.40	35.71
Nonparametric	0.57	1.40	4.12	7.28	35.78	0.35	0.78	2.97	5.52	28.70	0.53	1.24	3.79	6.88	33.72
DeepFillv2	0.57	1.43	4.14	7.34	35.93	0.34	0.78	2.96	5.50	28.95	0.53	1.24	3.92	7.04	33.45
Table A8: Evaluation on Synthetic Videos for Different Inpainting Methods for Forward Warp. Similar to Table 2, we evaluated and compared among different inpainting methods for both warp nearest and warp bilinear, including no inpainting (labeled as "None" in this Table), nonparametric inpainting, and DeepFillv2 [70].
Method	Inpainting	Motion Error ↓	Magnification Error ↓	SSIM ↑
Subpixel Test	Noise Test	Subpixel Test	Noise Test	Subpixel Test	Noise Test
0.04px	0.2px	1px	0.01x	1.0x	100.0x	0.04px	0.2px	1px	0.01x	1.0x	100.0x	0.04px	0.2px	1px	0.01x	1.0x	100.0x
Warp Nearest	None	36.20	3.74	2.25	0.71	0.89	207.76	1238.97	28.68	3.19	0.16	0.23	1.95	0.19	0.44	0.75	0.87	0.71	0.03
Nonparametric	10.60	3.01	2.12	0.65	0.91	221.32	209.10	22.61	3.12	0.16	0.23	1.90	0.77	0.92	0.97	0.97	0.85	0.18
DeepFillv2	10.66	3.00	2.13	0.66	0.91	220.66	206.04	22.39	3.13	0.16	0.24	2.06	0.72	0.91	0.97	0.97	0.85	0.14
Warp Bilinear	None	13.09	3.02	2.13	0.64	0.89	201.72	281.45	22.56	3.12	0.15	0.23	1.88	0.55	0.89	0.94	0.94	0.86	0.04
Nonparametric	10.61	3.02	2.12	0.61	0.90	226.37	209.35	22.61	3.10	0.15	0.23	1.68	0.78	0.93	0.97	0.98	0.90	0.19
DeepFillv2	10.56	3.02	2.12	0.62	0.90	225.30	207.24	22.64	3.11	0.15	0.23	2.32	0.77	0.93	0.97	0.98	0.90	0.09
Table A7 and Table A8 present a detailed comparison between the forward warp baselines without inpainting and with the two inpainting methods. In our evaluation of real-world videos, we observed a notable decrease in motion error when using inpainting methods in comparison to the warp baselines without inpainting, as expected. On the other hand, during noise tests, especially when using large noise factors, the warp baselines without inpainting exhibited better performance. Notably, we detected an increase in SSIM with noise factor increasing (see Table A6 for more details), for warp baselines with inpainting in noise tests. Our examination revealed that warp baselines produced distorted frames with significant noise factors and struggled to magnify motion when input motion was minimal. Consequently, the evaluation for forward warping baselines may not be able to reflect the actual performance in such cases. Therefore, given the slight advantage of nonparametric inpainting on real-world evaluations, we show the baselines with nonparametric inpainting in visualizations.
Discussion. The two forward warp methods, implemented with advanced techniques as the motion estimator and the inpainter, serve as powerful baselines for Lagrangian techniques, effectively establishing an upper performance limit for straightforward Lagrangian methods. The magnified frames produced through forward warping closely resemble the input frames, thus exhibiting favorable image quality due to their inherent implementation characteristics. Nevertheless, our method consistently outperforms or matches these baselines across various evaluation metrics. Upon qualitative assessment, our approach excels as well, particularly in complex scenarios involving occlusion and
Ours
Figure A5: Failure Cases. We report several typical failure cases of thin structures or background artifacts. See Section A8 for details.
Input (Guitar2)
Input (Camera)
disocclusion or challenging flow patterns. We provide one challenging case in Figure A4, where large motion and occlusion were involved. While the results from forward warping methods with/without inpainting suffer from handling the object boundary and background, our method is still capable to provide a satisfactory magnified frame.
A8 Failure Cases
While our method demonstrates favorable performance in most videos, there are cases where it falls short. One particular failure mode occurs when a video contains thin structures such as tree branches and guitar strings. In these cases it appears the flow network has incorrectly estimated the optical flow during training, causing our model to either "bleed" motion into the stationary background or fail to magnify the motion at all (see flower and guitar in Figure A5). The failure may be attributed to smoothness assumptions of flow estimators [50, 49, 23], and we expect results to improve as optical flow methods get more accurate. Another common failure is incorrectly magnifying background that has zero flow. This happens when small errors from the optical flow model are magnified, as shown in the camera sequence in Figure A5.