AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos
Authors: Yanze Wu
Xintao Wang
Gen Li
Ying Shan
Venue: NeurIPS 2022 Accept (2022)
Keywords: super resolution
animation video
degradation synthesis

Abstract: This paper studies the problem of real-world video super-resolution (VSR) for animation videos, and reveals three key improvements for practical animation VSR. First, recent real-world super-resolution approaches typically rely on degradation simulation using basic operators without any learning capability, such as blur, noise, and compression. In this work, we propose to learn such basic operators from real low-quality animation videos, and incorporate the learned ones into the degradation generation pipeline. Such neural-network-based basic operators could help to better capture the distribution of real degradations. Second, a large-scale high-quality animation video dataset, AVC, is built to facilitate comprehensive training and evaluations for animation VSR. Third, we further investigate an efficient multi-scale network structure. It takes advantage of the efficiency of unidirectional recurrent networks and the effectiveness of sliding-window-based methods. Thanks to the above delicate designs, our method, AnimeSR, is capable of restoring real-world low-quality animation videos effectively and efficiently, achieving superior performance to previous state-of-the-art methods.

AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos
Yanze Wu
Xintao Wang*1 Gen Li
Ying Shan'
ARC Lab, Tencent PCG
Plaatform Technologies, Tencent Online Video
Iyanzewu, xintaowang, enochli, yingsshan J @teeneent.c.mm .com
Abstract
This paper studies the problem of real-world video super-resolution (VSR) for animation videos, and reveals three key improvements for practical animation VSR. First, recent real-world super-resolution approaches typically rely on degradation simulation using basic operators without any learning capability, such as blur, noise, and compression. In this work, we propose to learn such basic operators from real low-quality animation videos, and incorporate the learned ones into the degradation generation pipeline. Such neural-network-baaee help to better capture distribution of real degradations. Second, a large-scale high-quality animation video dataset, AVC, is built to facilitate comprehensive training and evaluations for animation VSR. Third, we further investigate an efficient multi-scale network structure. It takes advantage of the efficiency of unidirectional recurrent networks and the effectiveness of bidding-windowbeae -based methods. Thanks to the above delicate designs, our method, AnimeSR, is capable restoring real-world low-quality animation vidcos effectively and efficiently, achieving superior performance to previous state-of-thee of-the-art methods. Codes and models are available at https::/githubb.co. TetmmeeennnnAAnniiiiii
basic operators could
1 Introduction
As a special category of video super-resolution (VSR) [5, 41, 47, 6], animation VSR aims at recovering high-resolution (HR) animation videos from their low-resolution (LR) counterparts. With the advent of the 4K/8K HD Era, people enjoy watching classic animation videos on HD devices. However, existing animation videos are mostly of low quality and have disturbing artifacts, which are usually caused by the production aging. lossy compression and transmission. Hence, highly desirable to develop practical animation VSR techniques to improve the resolution and quality of existing animation videos. Directly applying existing real-world SR methods (e.g., Real-ESRGAN [48], RealBasic [8]) produces unsatisfying and unnatural results with artifacts, such "hollow line" artifacts, annoying noises, etc, as shown in Fig. and Fig. 5.
In this paper, we thoroughly study and improve the key components in animation VSR.
Degradation synthesis process. Real-world animation VSR is blind SR task, whose LR videos have unknown and complex degradations, e.g., blur, noise and compression artifacts. Recent methods sort to simulate degradations close to the real-world ones as possible. The methods can be classified into two categories. One is to employ several classic basic operators together to form a complex degradation generation process [53, 48]. Those basic operators are usually "atomic" and have clear physical meaning, including blur filters, noise generators, JPEG/FEMPEG compressor, etc. However, relying on simple basic operators without any learning capability largely limits its synthesis ability of real-world degradations.
The other category is incorporate neural networks to synthesize LR samples with cycle consis- tency [57, 52, 13. 50] or self-supervision based on patch recurrence [58, 2]. Nevertheless, it is very
"Equal contributions.
Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Input
Real-ESRGAN
RealBasicVSR
AnimeSR (ours)
Figure 1: Comparisons of inputs, Real-ESRGAN [48], RealBasieVSR [8], and our AnimeSR results on real-world low-quality animation video frames. The AnimeSR model is capable of restoring more natural details while suppressing disturbing noises and artifacts.
challenging adopt one large neural network to learn the whole degradation process and the entire complicated degradation distribution. Thus, those methods only work for a very limited range of images and usually produce unpleasant artifacts.
In this work, we explore combining the above two categories and propose to learn basic operators for degradation synthesis. Specifically, we train tiny neural networks with 2 convolution layers to capture the main characteristics of real degradations. Those neural networks are then treated as basic operators, similar blur filters or noise generators, to form the whole degradation synthesis process. The neural-network-baseee basic operators are learnable and are able to synthesize real degradations that those classic ones cannot model. A simple illustration is shown Fig. 3.
Learning accurate and practical basic operators is still challenging, due to the lack of LR-HR paired data. Interestingly, observe that it is possible obtain pseudo HR samples for animation videos. For real LR video, we can resize the input into different resolutions and obtain several HR outputs by existing models (e.g., trained with classic operators). A satisfying output can be manually selected among those outputs. This is mainly due to the characteristics of animation videos, which usually consist of lines, sketches, and segments of smooth color pieces. More details are in Sec.
High-quality animation video dataset. The lack of a high-quality (HQ) animation video dataset impedes the development of animation VSR. Thus, we build large-scale HQ Animation Video Clip dataset, namely AVC. Unlike previous animation datasets that either consist of only single images or low-quality frames [43], AVC contains 553 HQ clips with total of 55,300 frames for training. Those clips are selected from various animation videos with different styles. Apart from the training set, also build real-world low-quality video test set from the Internet for practical evaluation. More descriptions and the statistics of AVC are provided Sec. 3.
Efficient network structure for VSR. For practical VSR, the unidirectional recurrent structure is usually adopted [41]. However, the unavailability subsequent frames in unidirectional structure hinders the exploitation of temporal information. Similar to [14], based on the efficient unidirectional structure, we further utilize the effectiveness of dliding-windon structures [47] in animation VSR. In order to fully exploit feature fusion different scales, we also adopt the multi-scale design inside unidirectional recurrent blocks to improve the performance.
The contributions are summarized as follows. (1) We propose to learn basic operators for degradation
synthesis, which better capture real-world degradations for animation videos. (2) A large-scalo scale
high-quality animation video dataset, AVC, is built for training and evaluations. (3) We investigate
an efficient multi-scale network structure, which takes advantage the efficiency of unidirectional recurrent networks and the effectiveness of Hiding-window-ber above improvements, our method, AnimeSR, capable of restoring real-world low-quality animation videos effectively and efficiently, surpassing recent state-of-the- thet-rr methods.
methods. (4) Equipped with the

---

AVC- Train
AVC-ReaILQ
RIT
6
AVC-Tran Roclution
ArD-IInis Reaolation
number or tramme 21
50C0
5000
ATD-11 Truin
AMS-Train
AVC RealLQ Resolution
AYC RealL Era
AVC-R
<1000p we1000p
az108Up
7200
3000
4500
1080p
20100s
19885s 19990s
China a Japan USA
20
32
40
53
70
hyperiQA score
Figure 2: Representative samples and statisties of AVC-Train and AVC-RealLQ sets. AVC-Train contains clips with larger resolutions and higher quality than the ATD-12K [43] dataset. AVC-RealLQ consists of diverse low-quality clips with various resolutions from different years and areas.
2 Related Work
Video Super-Resolution (VSR) has attracted increasing attention recent years. Different from image super-resolution [11, 24, 26, 17, 27, 29, 55, 49, 54, 32, 35], VSR further explores the temporal fusion and propagation scheme. Existing VSR methods can be roughly divided into two categories: sliding window-bar based methods [47, 5, 45, 19, 28] and recurrent- based methods [6, 7, 14, 20, 21, 18). Most previous methods focus on bicubic downsampling as the degradation. Recent works have started to address the real-world blind setting, such as DBVSR [39] and BSR [8]. In this work, we aim address the practical VSR for animation videos, which previous works have not investigated.
The degradation model used in existing SR methods can be divided into two categories: explicit degradation model [12, 31, 48, 53] and implicit degradation model [52, 13, 4, 34, 50, 56, 2. 10]. The explicit degradation model consists of basic degradation operators such as blur, noise and downsampling. It has developed from the early simple bicubic downsampling kernel to a classic sequence of blur, noise and downsampling, and finally to the current powerful high-order [48] and mandom-shuffies [53] models. The implicit degradation model attempts learn the real-world degradation process with a neural network. Due to the lack of real-world LR-HR pairs, recent methods either use unpaired data with cycle consistency [57, 52, 13, 4, 34, 50, 56] or self-supervision based on patch recurrence property [58, 2, 10] to train the network. However, these methods only work for limited range of images and usually produce unpleasant artifacts [30]. In this paper, we explore combining both the explicit and implicit degradation models.
Animation in this paper refers specifically to animated cartoons featured with exaggerated visual style. Various animation processing techniques using deep learning have been proposed in recent years. For example, animation style transfer [9], animation video interpolation [43], animation video generation for full-body characters [16]. However, existing animation datasets are usually of low quality and contain only single images or triplet frames. These shortcomings make them unsuitable for animation VSR, where HQ and long frame sequences are required. Thus, we build a large-scale HQ animation video clip dataset to facilitate the training and evaluation of VSR models.
3 Animation Video Clip (AVC) Dataset
To facilitate the training and evaluation of VSR methods on animation videos, we build a large-scale high-quality Animation Video Clip dataset, namely, AVC dataset. AVC is split into three sets. The training set, AVC- contains 553 high-quality clips with a total of 55,300 frames. The testing set, AVC-Test, contains 30 clips with total of 3,000 frames. The AVC-Test has similar quality to AVT-Train, but are excluded from the training set. In order to evaluate the methods in practical scenarios, we also build real-world testing set, AVC-RealLO, which consists of 46 low-quality clips from the Internet. Some examples of the AVC dataset are shown in Fig 2.
Dataset Collection of AVC-Train and AVC-Test. To ensure the high quality of the dataset, we first download large number of animation videos from the Internet and then manually select high-quality ones based on bit rate, frame resolution, and subjective quality. We use FFmpeg to extract frames
3

---

(a) Classic Basic Operators
(b)
Neural Networks
Learnable Basic Operators
cR. blur rescoling, nosse, JPEG compression
Leaming lange newors synthesine degrodations
Leaming tiny networiss basic oporators
Figure 3: Different ways to synthesize degradations. (a) Typical degradation synthesis uses pre- defined basic operators, such as blur, rescaling, noise and JPEG compression. (b) Neural networks are also employed to learn degradations. (c) We propose to learn tiny networks as basic operators.
with q: v setting to 1. We also adopt the hyperIQA [44], an image assessment quality algorithm. to calculate the quality score for each frame. Frames with low hyperIQA scores are discarded. To select the dynamic and meaningful scenes, PySceneDetect[]] is employed to detect and split scenes. Optical flow [40] is calculated to filter out static scenes. We keep a few clips with scene transitions, so that the trained models are more robust to
animation videos with scene transitions. To avoid redundancy and improve diversity, we only keep one clip for each video. The selection criterion weighted evaluation of hyperIQA scores and optical flow values. We always select high-quality frames with apparent motions. Following the practice in [38], the total frame number for each clip is set to 100. After this selection, we obtain 583 high-quality clips, and partition 553 clips for training and 30 clips for testing.
Dataset Statisties. The resolution and hyperIQA distributions of the AVC dataset are shown in Fig. 2. We also compare our AVC dataset with the recently proposed ATD-12K 12K [43] to demonstrate the superiority of the AVC dataset. Our AVC dataset contains clips with larger resolutions than ATD-12K HyperIQA scores reflect the perceptual quality, whose values are between 1 and 100. The higher the scores are, the better the quality is. We can observe that the AVC dataset also has higher quality than ATD-12K. Besides, recent VSR methods (e.g.. Basic [6]) require long sequences (e.g.. 15 frames) during training to better leverage long-term propagation. However, ATD-12K with only triplet frames is not suitable for the VSR task.
AVC-ReaILQ. To fully evaluate the model's ability practical scenes and assess the generalizability of VSR methods, we also construct a real-world low-quality video test set from the Internet, namely, AVC-ReaILQ. Those videos are downloaded from various video sites such Bilibili, Youtube, and Tencent Video. The different resolutions, contents, and genre are considered for the diversity. We collect 46 real-world low- low-quality clips, with each clip containing 100 frames. The representative samples and statistics are shown in Fig. 2.
4 Methodology
4.1 Learning Basic Operators for Degradation Synthesis
Real-world animation video super-resolution (VSR) aims to restore high-resolution videos from low-resolution ones with unknown and complex degradations, e.g., blur, noise and downsampling. Due to the lack of LR-HR training pairs, recent works sort to design degradation models as close to the real-world degradations as possible. Then, LR samples are synthesized from available HR samples using the degradation model. The degradation model can be described as an n-step degradation process and formulated as:
x= D")" (D, .... D2 Di) (3))
where y is an HR input, æ is the synthetic LR output, and D; is the i-th basic degradation operator.
The classical degradation model is four-step process, in which basic operators include blur filters, noise generators, rescaling, JPEG/FFMPEG compressor, etc. High-order degradation process [48] and shuffling strategy [53] are then proposed to expand the degradation space, so that it can cover more practical degradations. However, the degradation process still involves those classic basic operators only. We denote this category as Basi c0p-Onl y,, illustrated in Fig. 3 (a). The basic operators are "atomic" and do not have any learning ability, which essentially limits their synthesis ability of real-world degradations. Another category adopts large neural networks (NN) and adversarial learning to synthesize LR samples. We denote this category as NN- Only, as shown in Fig. 3 (b). However, it is challenging employ one large neural network to learn the whole degradation process and the entire complicated degradation distribution. As result, those methods only work for a limited range of images and usually produce unpleasant artifacts.

---

100 スス
X
100
J5
49
100
X
6%
100
X
6B
100
7U
77
100
100
10
10
100
10
Inpat
x1
x0.9
x0.7
x0.5
x0.3
SR resudts under different rescaling factor factor of inputs
Figure Illustration of the "input-rescaling strategy We resize the inputs with different rescaling factors, Le., from x1 to x0.3, and generate the SR results using VSR model trained with BasicOP-OnIy degradations. We can observe "hollow line" artifacts, annoying noises and unwanted textures for x1 rescaling. The artifacts gradually reduced as the input resolution decreases. Rescaling the inputs by x0.5 produce a satisfying output. Zoom in for best view
As shown in Fig. (c), we propose to learn basic operators for degradation synthesis. Unlike NN-Only methods train tiny
Degradation Space
Degradation Space
that large network, neural networks with 2 or 3 convolution layers to capture the main characteris- ties of real degradations. Those neural networks are then incorporated with the classic basic operators to form the degra- dation model. The neural operators are learnable and are able to synthesize real degradations that cannot modeled by
Classic
Basic Operators
Learnable Hasic Operators
Figure 4: sic operators. The learnable basic operators expand the degradation space with more real degradations. those classic ones. As illustrated in Fig. 4, with the learnable basic operator, the degradation space is largely expanded and can cover more real degradations.
The degradation space "spanned" by ba-
How to obtain learnable basic operators? Different to NN-Only methods that train with adversarial learning and unpaired data, we train Icarnable basic operators with LR-HR pairs supervised manner to learn the mappings from HR to LR. Such strategy is more controllable and introduce fewer artifacts. Yet, it challenging to get LR-HR pairs of real-world low-quality (LQQ videos for training. Our analysis below shows that it is possible to obtain pseudo HR samples for real animation videos.
For real LQ animation videos, we can easily obtain the preliminary SR results using a VSR model trained with commonly-used Bas cOP degradations, shown in the second columns of Fig. 5. As expected, the outputs are unsatisfying. We can observe "hollow line" artifacts, annoying noises and unwanted textures in the second columns of Fig. 5. We then resize the inputs with different rescaling factors, e.g., from x to 0.... The rescaled inputs are sent to the same VSR model, the corresponding SR results are presented in Fig. (the third column to the last column). It can be observed that the artifacts are gradually reduced as the input resolution decreases. But too large downscaling factor on inputs leads detail/information loss. Among them, rescaling the inputs by x0.5 on those video samples produce good trade-off between artifacts elimination and detail loss. Thus, satisfying output can be manually selected as the pseudo HR. We call this simple yet effective strategy as "input-rescaling strategy Note that the best rescaling factor can vary for different real-world animation videos. More details about the "input-rescaling strategy" can be found in the supplementary material.
5

---

LR
Recurrent Block
R
Fusson
ResBIk
ResBik
ResBik
1x2
L.R,
Recurrent Block
SR;
ResBIk
ResBik
ResBik
X2
J x2
ResBik
ResBik
1x
1Re4s
Recurrent Block
S
(a) AnimeSR architecture
(b) Multi-scale recurrent block
Figure 6: The network structure of AnimeSR. (a) AnimeSR combines the efficiency of unidirec- tional recurrent networks and the effectiveness of sliding-widow-beee widow-based methods. (b) The recurrent block employs a multi-scale design to fully exploit the capability for animation videos.
Why the "input-rescaling strategy" works for animation video? The "input-rescaling strategy" is able to eliminate artifacts while not affecting the details under proper rescaling factor. We hypothesize that: 1). the degradation of real LQ videos probably outside the synthesized degradation space "spanned" by classic basic operators. Rescaling the inputs changes the degradation and may narrow its gap towards synthesized degradation space. Besides, downscaling also makes the LR samples look "cleaner", which is also pointed in [13]. 2). Animation videos usually consist of lines, sketches, and segments of smooth color pieces, which have very different characteristics from natural videos. The particular characteristics of animation videos lead to a phenomenon that rescaling within a limited range will not have much influence on details. This strategy cannot be directly applied in natural videos, it will largely affect the textures.
Details of training learnable basic operators. We pick out several representative real-world LQ animation videos to train learnable basic operators. We select LQ videos, on which the VSR model performs poorly on the original scale, but produce satisfactory results on a suitable rescaling factor. Then, determine the best rescaling factor of inputs for each video. After that, about 2000 frames are collected for one LQ video. Similar to the collection pipeline of the AVC dataset, the motions and scene diversity are considered during the collection. We then resize those frames with the best rescaling factor, feed them into the VSR networks, and finally obtain pseudo HR samples.
The LR- -R-pseudo HR pairs are then used to train the learnable basic operators. The tiny neural operator lightweight and is composed of three 3 x 3 convolution layers with the hidden channel dimension of 64. LeakyReL.U activations are employed between convolution layers. The training objective of neural operators is the same as that for the main network structure, as described in Sec. 5.
Discussions. 1). We observe that learned operators have strong generalization. An operator trained on one LQ video can also be effective for other LQ videos. This property largely improves its practicability. We hypothesize that the captured real degradations probably have commonality, and cannot be well modeled by classic basic operators. 2). Unlike the classical basic operator with a clear physical meaning for each one, what the neural operator learns does not have a clear definition. It may capture mixture of blur and noise, reflecting the main characteristics of the real degradation in LQ videos. The visualization of leaned basic operators is demonstrated in Sec. 5.3. 3). In this paper, we train three learnable basic operators from three different LQ videos respectively and put them into a pool. During each training iteration, we randomly select one from the pool and directly incorporate into the degradation generation pipeline. This is a simple incorporation design of learnable basic operators. More schemes can be explored in future works.
4.2 Network Structure
The network structure in practical animation VSR requires a good balance between performance and efficiency. Recent practical models such as BSRGAN [53], Real-ESRGAN [48] and RealBa- sicVSR [8] usually adopt a very large network, whose processing is time-consuming and resource- intensive. This shortcoming becomes more severe when super-resolving existing videos to 4K/8K resolutions. For practical VSR, the unidirectional recurrent structure is usually adopted [41]. Yet, the unavailability of subsequent frames hinders the exploitation of temporal information. There- fore, based on the efficient unidirectional structure, we further employ the simple sliding- winddw structure [47].
The overview is depicted in Fig. (a). The upsampling operation is performed at the last layer with one sixel-shuffle layer [42] to obtain SR outputs, so that most of the calculation is conducted in the LR feature space. At the time i, the recurrent block receives the information from the previous hidden state S-1 of the recurrent block, and the SR output of time t 1. The output SR,11 is

---

of high resolution and is downsampled to match the spatial size of the recurrent block using pixel-unshuffle layer. The recurrent block also receives sequence of 1L LR, LR I frames, as sliding-window- window-basse based methods do.
As analyzed in Sec. 4.1, rescaling inputs of animation videos achieve better balance of detail enhancement and artifacts elimination. Thus, we also adopt the multi-scale design inside the unidirectional recurrent block. Besides, the multi-scale structure can also help to fully exploit feature fusion at different scales. As shown in Fig. 6 (b), we adopt 10 residual blocks in the recurrent block. We employ three scales, e.g., x1, x0.5 and x0.25. We allocate 5. 3 and 2 blocks for those three scales, respectively. We do not employ optical flow in our AnimeSR, as we empirically find that optical flow does not bring apparent visual improvements. Besides, calculating optical flow also slow down the training and inference.
Experiments
5.1
Training Details
The training process consists of two stages. In the first stage, we train models with the LI loss for
300K iterations. In the second stage, we finetune the models for another 300K iterations with L1
loss, perceptual loss [23]. and GAN loss [27]. whose loss weights are all set to 1. The discriminator is
three-scale PatchGAN [22, 46] with spectral normalization [37]. We use the Adam optimizer [25] with a learning rate of x 10 14 for the first stage and a learning rate of 1 x 10 4 for the second stage. We set the batch size per GPU, frame sequence length, and patch size of the HR frames to 4, 15, and 256, respectively. All the training is performed with PyTocc on four NVIDIA A100 GPUs in an internal cluster.
To obtain the learnable basic operators, we first train a VSR model with a degradation model that only contains the classic basic operators (Fig. 3 (a)). Then we utilize this model and "input-rescaling strategy" to generate the pseudo HR samples. The training strategy for the learnable basic operators is almost the same mentioned above, except that the batch size and iteration set to 16 and 100K. The final VSR model is trained with a degradation model containing both classic basic operators and learnable operators. Specifically, the degradation model can be formulated as:
(2) where y is an HR frame sequence, æ is the synthetic LR frame sequence. LBO denotes the Learnable Basic Operator. Blur represents blur filters, and we employ isotropic and anisotropic Gaussian kemells with probability of {0.7, 0.3). Noise denotes the Gaussian color noise and Gaussian gray noise with a probability of (0.5, 0.5). Down denotes the downscaling operation with a scale factor of Finally, we use FFmpeg perform video compression. More details about the hyper-parameters are provided in the supplementary materials.
x = (y) = [FFpeeq LBO Down o LBO o Down o Noise (l)r)
5.2 Comparisons with Previous Methods
We compare our method with recent statc-of-the-art methods for real-world SR, including BSR- GAN [53]. Real-ESRGAN [48] and RealBasicVSR [8]. For a fair comparison, we finetune their officially released models on our proposed AVC-Train dataset. We also provide the comparisons before and after finetuning in the supplementary material.
Quantitative comparisons. All methods are evaluated on the real-world AVC-RealL dataset. As ground-truths are not available for real-world LQ animation videos, use no-reference image quality assessment (NR-IQA) metrics NIQE [36] and MANIQA [51] to evaluate the methods quantitatively. NIQE is widely used for the evaluation of real-world SR models [53, 48, 8]. is based on natural scene statistics and hand-crafted features. MANIQA is the winner solution of the NTIRE 2022 NR-IQA challenge [15], it is a learning-based approach and employs a powerful VIT to extract features. The comparison results are shown in Tab. 1. We train two AnimeSR models with one and three learnable basic operators. Our method achieves better NIQE and MANIQA scores with significantly smaller model size and faster processing speed (i.e., 13.7x x faster than BSRGAN and 5.9x faster than RealBasieVSRR) The high efficiency makes our AnimeSR more practical animation VSR model. As many previous works [3, 33, 53] have pointed out, NR-IQA metrics are not always consistent with visual quality, especially on the finer scale. Empirically, we find that the recent MANIQA [51] is more consistent with the perceptual visual quality than NIQE.
7

---

BSRGAN* Real-ESRGAN" RealBasicVSRR* AnimeSR(ours) Qualitative comparisons. *** denotes fine-tuning on our AVC dataset. Our AnimeSR could recover cleaner lines and produce more visual-pleasing outputs, while having fewer artifacts. AnimeSR can also restore natural textures in blurred background, rather than impair those textures (4th row). Zoom in for best view
Input
Figure 7:
Table 1: Quantitative comparisons on AVC-RealLQ. LBO denotes the Learnable Basic Operator. denotes fine-tumm fine-tuning on our AVC dataset. The runtime is computed on Nvidia A100 with the LR size of 480 x 640. The time is not included.
BSRGAN [S3]
Real-ESRGAN [48]
RealBasie [8]
AnimeSR LBO
AnimeSR
ICCV21
ICCVW 27
CVPR
LB0
Params (M))
16.7
16.7
6.3
1.5
1.5
Runtime (ms).
439
439
190
32
32
NIQEI
8.6369
8,3220
8,6393
8,1444
8,7088
MANIQA
0,3753
0.3782
0,3602
0.3763
0,3832
Qualitative comparisons. We also show qualitative comparisons real-world clips in Fig. 7. It is observed that our AnimeSR can recover cleaner and sharper lines with fewer artifacts (the 1st row and 3rd row), and produce more natural outputs in the fountain and water in the 2nd row, while previous methods generate wired lines and patterns. For the fourth row in Fig. 7, the input frame has blurred background with shallow depth of field. Previous methods either over-sharpen the background or impair the textures. While the AnimeSR able to retain the overall naturalness and coherency. More visual comparisons are in the supplementary materials.
video
5.3 Ablation Studies and Discussions
Unless specified otherwise, we adopt one learnable basic operator (LBO) in ablations. Empirically, find that the recent MANIQA [51] is consistent with the perceptual visual quality than NIQE. Thus, we report the MANIQA score for ablation studies.
Influence of different animation datasets. We compare models trained with our AVC dataset and ATD-12K dataset. The model trained with AVC dataset could produce outputs with higher quality (Tab. 2) and cleaner lines, shown in Fig. 8 (a).
Effectiveness of the learnable basic operators. We compare the following variants. BasicoP-ON denotes the VSR model trained without LBO. For NN-Only, employ the degradation model proposed in DSGAN-SR [13]. DSGAN-SR employs a large network to learn real-world degradations in unpaired manner. The network has 594K K parameters and is significantly larger than our tiny learnable basic operator. We re-train DSGAN-SS for fair comparison. We also compare
AnimeSR with one LBO and three LBOs. The qualitative comparison is shown in Fig. 8 (b). We can observe that Basi BasicOP Only produces "hollow line" artifacts while NN-Only introduces extra artifacts around lines. Our method can clearly sharpen lines without annoying noises. Quantitative results Tab. also show the superiority our degradation model with LBOs.
Benefits of multi-scale architecture. The effectiveness of the "input-rescaling strategy" motivates us to employ multi-scale architecture. We compare our network structure with the single-scale one to validate its effectiveness. Both architectures have the same number of parameters. From Fig. 8 (c), we can find that the multi-scale architecture could produce cleaner lines and fewer artifacts than the single-scale architecture.

---

Input
ATD-12K
AVC-Train
Input
BasicOP-Only
NN-Only
Ours
(a) Training Set Ablation
(b) Degradation Model Ablation
Input
single-scale
multi-scale
LBO-1
I.BO-2
L.RO-3
I.O pool
(c) Multi-scale Ablation
(d) LBO Ablation
Figure 8: Visual comparisons of different ablation studies. Zoom in for best view
Table 2: Ablation studies the dataset, degradation model, multi-scale structure and the learnable basic operators (LBO). We report the MANIQA scores for comparison.
Training Ablation
Degradation Model Ablation
ATD-12K
AVC-Train
Basicor-Cnly
NN-only
ours w/ LBO
ours w/ LBO
MANIQA
0,3309
0.3763
0.3554
0,3034
0,3763
0.3832
Multi-scale Ablation
1,00 Ablation
single-scale
multi-scale 0.3763
LBO-1
LBO-2
LBO-3
pool
MANIQA
0.3283
0.3763
0.3803
0.379
0.3832
Benefits of LBO pool. We learn three learnable basic operators (we use LBO-1, LBO-2, and LBO-3 denote the three different LBOs) from three representative LQ videos and then put them into pool for random selection during training. We compare such strategy with a single LBO. It is observed from Fig. (d) that using single LBO probably leads ghosting artifacts and unclear lines in the curtains, while LBO pool can produce clean lines and sketches. Employing the LBO pool also achieve higher MANIQA scores from Tab.
Visualizations of what the learnable ba- sic operators learn. We visualize the LQ patches synthesizing from the same HQ image using different basic operators. As shown in Fig. 9, the same HQ input patch goes through five differrnt basic degradation operators, generating five LQ patches. These degradation operators consist of two classic basic operators (i.e., Gaussian blur and Gaus- sian noise) and three learnable basic opera- tors (i.e., LBO-1, LBO-2, and LBO-3). It observed that the degradations learned by LBO are pretty different from classic basic operators, e.g., Gaussian blur and Gaussian noise. It seems that LBO captures mixture of blur and noise (here blur and noise stand for vague human perception). We can also observe color jitter around lines in the LBO-1 neural operator, which common in highly-compressee LQ videos. Besides, the three LBOs learn different degradations. The diversity LBO makes it meaningful to combine multiple LBO during training.
ingut
Gaussian blur
Gaussian noise
LBO-1
LB0-2
LBQ-3
Figure 9:
LQ patches generated using basic opera- tors. Learnable basic operators capture diverse degra- dations that are different from the predefined ones (e.g.. blur and noise). Zoom in for best view
Limitations and Discussions. Our work has several limitations. 1) The learning of basic operators in supervision manner heavily relies on the "input-rescaling strategy". It fully exploits the characteristics of animation videos but cannot be directly applied to videos of natural scenes. 2) We are interested in building larger pool learnable basic operators from more real I.Q videos. The complementarity and substitutability of these operators remain be explored.

---

6
Conclusion
This paper gives a comprehensive study of the practical animation video super-resolution task. To facilitate the training and evaluation of animation VSR, we build a large-scale HQ animation video dataset named AVC. We also propose learn basic operators in the degradation generation pipeline to better capture the distribution of real-world degradations. Learning of those neural operators is made possible by the effective "input-rescaling strategy" for real low-quality animation videos. We further investigate an efficient multi-scale network structure. Extensive experiments demonstrate the capability of our proposed method, AnimeSR, in restoring real-world low-quality animation videos, outperforming previous state-of-the-an of-thc-art methods.
References
[1] https://g thub. com/ Brea www.akth wtw.kk//wwoooogg/yyyeeeetteeeeee 2022. 4
[2] Sefi Bell-Kligler, Assaf Shocher, and Michal Trani. Blind super-resolution kernel estimation using an internal-gan. In NewrIPS, 2019. 1,3
[J] Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor, The 2018 pirm challenge on pereeptual image super-resolution. In ECCV Workshops, 2018. 7
[4] Adrian Bulat, Jing Yang, and Georgios Tzimiropoulos. To learn image super- repeerresolution, gan to leam
how to do image degradation first. ECCV. 2018. 3
[5] Jose Caballero, Christian Ledig. Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. Real-time video super-resolution with spatio-temporal networks and motion compensation.
In CVPR, 2017. 1,3
[6] Kelvin CK Chan, Xintao Wang. Ke Yu, Chao Dong, and Chen Change Loy. Basievsr: The search for essential components in video super-resolution and beyond. In CVPR, 2021. 3.4.4
[7] Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Lay. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment. In CVPR, 2022. 3
[8] Kelvin CK Chan, Shangehen Zhou, Xiangyu Xu, and Chen Change Loy. Investigating tradeoffs in real-world video super-resolution. In CVPR, 2022. 1, 2, 3, 6, 7,
Chen, Gang Liu, and Xin Chen. Animegan: novel lightweight gan for photo animation. In ISICA, 2019. 3
[10] Xi Cheng. Zhenyong Fu, and Jian Yang. Zero-shot image super-resolution with depth guided internal degradation learning. ECCV, 2020.
[1]] Chao Dong, Chen Change Loy. Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In ECCV, 2014.
[12] Michael Elad and Arie Feuer: Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE TIP, 1997. 3
[13] Manuel Fritsche, Shuhang Gu, and Radu Timofte. Frequency separation for real-world super-resolution.
In ICCV Workshops, 2019. 1, 6,
[14] Dario Fuoli, Shuhang Gu, and Radu Timofte. Efficient video super-resolution through recurrent latent
space propagation. In ICCV Workshops, 2019. 2,
[15] Jinjin Gu, Haoming Cai, Chao Dong, Jimmy S Ren, Radu Timofte, Yuan Gong, Shanshan Lao, Shuwei Shi, Jiahao Wang, Sidi Yang, al. Ntire 2022 challenge on perceptual image quality assessment. In CVPR
Workshops, 2022. 7
[16] Koichi Hamada, Kentaro Tachibana, Tianqi Li, Hiroto Honda, and Yusuke Uchida. Full-body high- resolution anime generation with progressive structure-conditional generative adversarial networks. In
ECCV Workshops, 2018, 3
[17] Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. Deep backprojection networks for super- resolution. CVPR, 2018.
[18] Yan Huang, Wei Wang. and Liang Wang. Bidirectional recurrent convolutional networks for multi-frame
super-resolution. In NewrIPS, 2015. 3
[19] Yan Huang, Wei Wang. and Liang Wang. Video super-resolution via bidirectional recurrent convolutional
networks. IEEE TPAMI, 2018. 3
[20] Takashi Isobe, Xu Jia, Shuhang Gu, Songjiang I.i, Shengjin Wang, and Qi Tian. Video super-resolution with recurrent structur-detail network. In ECCV, 2020.
[21] Takashi Isobe, Fang 7hu, Xu Jia, and Shengjin Wang. Revisiting temporal modeling for video super-
resolution. RMVC, 2020. 3
[22] Phillip Isola, Jun-Yan 7hu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017. 7
[23] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
10

---

[24] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016. 3
[25]
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCLR, 2015. 7
[26]
Wei-Sheng Lai, Jia-Bin Huang. Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In CVPR, 2017. 3
[27] Christian I Ledig. Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-rcalistic single image super-resolution using a generative adversarial network. In CVPR, 2017. 3,
[28] Wenho Ii, Xin Tao, Taian Guo, Lu Qi, Jiangbo Lu, and Jiaya Jia. Mucan: Multi-correspondence aggregation network for vidco super-resolution. In ECCV, 2020. 3
[29] Bee I. Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual
networks for single image super-resolution. In CVPR Workshops, 2017. 3
[30] Anran Liu, Yihao Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Blind image ruper-nesolution A survey and beyond. arXiv:2 107.03055, 2021. 3
[31] Ce Liu and Deqing Sun. On bayesian adaptive video super resolution. IEEE TPAMI, 2013. 3
[32] Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang. Non-local recurrent network for image restoration. In Neur/PS, 2018. 3
[33] Andreas Lugmayr, Martin Danelljan, and Radu Timofte. Ntire 2020 challenge on real-world image
super-resolution: Methods and results. In CVPR Workshops, 2020. 7
[34] Shunta Maeda. Unpaired image super-resolution using pseudo-supervision. CVPR, 2020.
[35] Yiqun Mci, Yuchen Fan, Yuqian Zhou, Lichao Huang. Thomas S Huang, and Honghui Shi. Image super- resolution with cross-scale non-local attention and exhaustive self-exemplars mining. In CVPR, 2020.
[36] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer.
IEEE Signal Processing Letters, 2012. 7
[37] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018. 7
[38] Scungjun Nah, Sungyong Baik, Scokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lec. Ntire 2019 challenge on vidco deblurring and super-resolution: Dataset and study. In CVPR Workshops, 2019. 4
[39] Jinshan Pan, Haoran Bai, Jiangxin Dong, Jiawei Zhang, and Jinhui Tang. Deep blind video super-resolution. In ICCV, 2021. 3
[40] Anurag Ranjan and Black Michacl. Optical flow estimation using spatial pyramid network. In CVPR,
2016. 4
[41] Mchdi S M Sajaadi, Raviteja Vemulapalli, and Matthew Brown. Frame-rccurrent vidco super-resolution.
In CVPR, 2018 2,
[42] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Toiz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
[43] Li Siyao. Shiyu Zhao, Weijiang Yu. Wenxiu Sun, Dimitris Metaxas. Chen Change Loy, and Ziwei Liu. Deep animation video interpolation in the wild. In CVPR, 2021. 2, 3,4
[44] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge. Jinqiu Sun, and Yanning Zhang. Blindly assess
image quality in the wild guided by self-adaptive hyper network. In CVPR, 2020. 4
[45] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu. Tdan: Tenporally-deformable alignment network
for video super-resolution. In CVPR, 2020. 3
[46] Ting-Cli Chun Wang. Ming- Yu Liu, Jun-Yan Zhu, Audrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018. 7
[47] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In CVPR Workshops, 2019. 1, 3,
[48] Xintao Wang, Lianghin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super- resolution with pure synthetic data. In ICCV Workshops, 2021. 1, 2, 3, 4, 6, 7,8
[49] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In ECCV Workshops, 2018. 3
[50] Yunxuan Wei, Shuhang Gu, Yawei Li, Radu Timofte, Longeun Jin, and Hengjie Song. Unsupervised real-world image super resolution via domain-distance aware training. CVPR, 2021. 1,
[51] Sidi Yang. Tianhe Wu, Shuwei Shi, Shanshan Lao Yuan Gong, Mingdeng Cao, Jiahao Wang. and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In CVPR Workshops, 2022. 7,8
[52] Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong, and Liang Lin. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In CVPR, 2018. 1,3
[53] Kai Zhang, Jingyun Liang, Lue Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. In ICCV, 2021. 1, 4, 6, 7,8
11

---

[54]
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong. and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, 2018. 3
[55]
Yulun Zhang, Yapeng Tian, Yu Kong. Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR, 2018. 3
[56] Yuanbo Zhou, Wei Deng, Tong Tong, and Qinquan Gao. Guided frequency separation network for
real-world super-resolution. In CVPR Workshops, 2020.
[57]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017. 1,
[58] Maria Zontak and Michal Trani Internal statisties of single natural image. In CVPR, 2011. 1,3
12


Reviews:
[{"review_id": "zEusZnOSHbZ", "reviewer": "Reviewer_Pf9p", "summary": "This paper aims to improve blind video super-resolution (VSR) for animation videos. It combines two categories of degradation synthesis processes (classic basic operator and neural networks) to overcome both limitations. Besides, the authors build a new high-quality animation video dataset for animation VSR. Moreover, the paper designs an efficient network structure for VSR based on sliding window structures and multi-scale design. The experiments reveal that the proposed model outperforms other state-of-the-art models and validate the effectiveness of the model design.", "questions": "1. The author should present how to use the “input-rescaling strategy” in detail and discuss the time-consuming issue.\n2. The author should differentiate the proposed method from FRVSR, especially the sliding window design.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It is interesting and seems promising to combine classic basic operators and neural networks together and learn basic operators by neural networks.\n2. The authors build a large-scale high-quality animation video dataset for animation VSR.\n3. There are plenty of experiments in the paper to validate the effectiveness of the model design. The experiments demonstrate that the proposed methods perform better than other state-of-the-art models and the combination of classic basic operators and neural networks is beneficial.\n\nWeaknesses:\n1. The use of the “input-rescaling strategy” is not demonstrated well. How do the users manually select the pseudo HR? It seems time-consuming to do it.\n2. The intuition behind sliding window structure and multi-scale design is not demonstrated well. How to distinguish it from FRVSR? FRVSR is also based on sliding windows.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "n5uFR54A28x", "reviewer": "Reviewer_7uTp", "summary": "This paper proposes an animation video super-resolution method. \nFirst, the authors build the proposed AVC dataset by collecting low-quality videos and by generating pseudo-HR videos. Pseudo-HR videos are obtained by rescaling low-quality videos and super-resolving them with a conventional SR method. \nSecond, the degradation of animations are modeled by the combination of conventional corruption artifacts and learnable convolutional layers. The authors train their degradation model with the AVC dataset LR & pseudo-HR pairs.\nThird, the proposed model architecture, window-based RNN is trained by the pseudo-HR dataset and the generated LR videos.\n", "questions": "L172-174 and Fig. 5. \nI don’t exactly understand what the process of input-rescaling strategy is. Is input low-quality video frame downsampled and then super-resolved from a BasicOP-Only method?\n\nFigure 9\n\nIs this a result of equation (2)-like cascade of blur, noise, and LBO?\nIn equation (2), does two LBO share weights?\n\nL318 \n\nBlur and noise are explicitly modeled in equation (2) even without LBO. Why does LBO learn to blur and noise images?\n\nL306\n\nIs one LBO learned per a video?", "limitations": "Limitations discussed in the manuscript.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\nThis paper tackles the animation super-resolution problem by observing the distribution of animation and natural videos differ. The observation is validated by comparing the SoTA methods and the proposed method on animations.\nIn order to handle the unknown degradation artifacts that are hard to model, the proposed method combines the known operations (blur, noise, FFMPEG) and learnable operations (conv layers). This is a novel and effective approach.\n\nAlso, the authors construct pseudo-HR data from their observations which exhibits validity through experimental results.\n\n[Weaknesses]\n\nHowever, I wonder what the main principle is in the collected & generated dataset.\nThe video selection seems to be manual (which is fine) and the rescaling factor selection seems to rely on human perception. Were there any analyses that are not shown in the paper?\nThe authors use NIQE and MANIQA metrics. Any correlations with those metrics?\nIf the selection process is manual, it doesn’t seem to be a reproducible and scientific approach.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GUm5Lk-FcY", "reviewer": "Reviewer_UxQQ", "summary": "This paper aims to develop an effective and efficient real-world video super-resolution (VSR) method to restore real-world low-quality animation videos. The authors first proposed to learn basic operators with tiny neural networks (with 2 or 3 conv layers) to incorporate the learned networks into the degradation synthesis process. Secondly, the authors propose a new dataset AVC which is a large-scale high-quality animation video dataset for training and evaluation. The authors perform experiments on AVC dataset and demonstrate superior performance compared to state-of-the-art models.", "questions": "1. Does the proposed input-rescaling strategy only work for animation videos? Since the authors anticipate that the input-rescaling strategy will not work well for natural videos due to textures, ablation studies on a texture could be helpful to quantitively investigate the scalability of this work.\n\n2. How to determine the best rescaling factor of inputs for each video? Since the optimal rescaling factor varies on each video, pseudocode to determine the rescaling factor can help understand the overall process.\n", "limitations": "Societal Impact: There is no significant negative societal impact of this research.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: Instead of using one large neural network for the whole degradation process or classic basic operators without any learning capability, this paper proposes a novel method to learn basic operators with tiny neural networks. Meaningful discussion is also a strength of this paper (e.g. discussion on input-rescaling strategy, visualization of learned LBOs). Further, this paper presents a new dataset that contains more than a total of 50,000 frames, which is one of the main contributions of this paper.\n\nWeakness: The writing can be improved. Section 4.2 is particularly confusing. The novelty of the approach seems limited. In particular, many components presented in this work are simple modifications from existing work (e.g. combining the classic basic operators and existing neural-network-based operators). Further, the proposed supervised manner of training heavily relies on an input-rescaling strategy, which restricts the scalability of work to the animation videos.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept