Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 17:19:17

Abstract: Hand-drawn 2D animation workflow is typically initiated with the creation of sketch keyframes. Subsequent manual inbetweens are crafted for smoothness, which is a labor-intensive process and the prospect of automatic animation sketch interpolation has become highly appealing. Yet, common frame interpolation methods are generally hindered by two key issues: 1) limited texture and colour details in sketches, and 2) exaggerated alterations between two sketch keyframes. To overcome these issues, we propose a novel deep learning method - Sketch-Aware Interpolation Network (SAIN). This approach incorporates multi-level guidance that formulates region-level correspondence, stroke-level correspondence and pixel-level dynamics. A multi-stream U-Transformer is then devised to characterize sketch inbewteening patterns using these multi-level guides through the integration of self / cross-attention mechanisms. Additionally, to facilitate future research on animation sketch inbetweening, we constructed a large-scale dataset - STD-12K, comprising 30 sketch animation series in diverse artistic styles. Comprehensive experiments on this dataset convincingly show that our proposed SAIN surpasses the state-of-the-art interpolation methods. Our code and dataset will be publicly available.

Main paper: 
 introduction: 
 Hand-drawn 2D animation is extensively used in the animation industry for unique artistic expression, emotional depth and versatility. Notably, Your Name (2016) and Big Fish & Begonia (2016) achieved enormous success in recent years. The hand-drawn 2D animation workflow typically involves three key stages: sketching keyframes, inbetweening keyframes to produce intermediate sketch frames (i.e., inbetweens), and colorization to produce the final, full-color animations. The meticulous creation of inbetweens is crucial for achieving a smooth animation with lifelike motion transitions, effectively conveying the intended story or message. For a feature-length animation created through this process, the Figure 1 : Illustration of sketch inbetweening: the limitations of relevant methods in comparison to our SAIN, including LDFI [17] for sketches, EISAI [3] for color animations, and DQBC [40] for videos.sheer volume of required inbetweens can be staggering [33] , making it a highly specialized and labor-intensive task and serving as a limiting factor in overall animation productivity.To streamline the process of 2D sketch animation production, various studies have focused on the automatic synthesis of inbetweening sketch frames, which take two consecutive sketch keyframes as input and produce interpolated intermediate sketch frames (i.e., inbetweens) as output. These methods can be categorised into stroke-based and image-based. The stroke-based methods often rely on a labor-intensive pre-processing step for sketch vectorisation [29, 35, 38] , whilst subpar vectorisation quality can negatively impact the final outcomes. Image-based methods treat sketch frames as bitmap images, applying conventional image or video interpolation algorithms. However, they commonly face two significant challenges: 1) the absence of texture and color details in sketch frames, hindering reliable image-based inbetweening correspondence, and 2) exaggerated changes due to substantial object movements between two consecutive sketch keyframes [17] . As a result, when image-based methods, especially devised for videos [40] and colour animations [3, 30] , are applied to sketch interpolation, they invariably introduce various artifacts into the produced interpolated frames. These discrepancies can adversely affect the continuity and quality of the animation produced. As shown in Figure 1 , LDFI [17] proposed for sketch interpolation generates broken strokes due to the missing sketch keypoint correspondence, while EISAI [3] proposed for interpolating color animation frames and DQBC [40] for video interpolation introduce blurriness (ornaments) and artifacts (e.g., distortion in face regions).Therefore, in this study, we propose a novel deep learning method for sketch interpolation, the Sketch-Aware Interpolation Network (SAIN), to comprehend and model the intricate and sparse patterns found in hand-drawn animation sketches. SAIN adopts a sketchaware approach that integrates multi-level sketch-related guidance through three distinct aspects: 1) pixel-level dynamics at a fine level with a bi-directional optical flow estimation module, 2) stroke-level correspondence with a stroke matching and tracking mechanism for obtaining stroke keypoint traces and 3) region-level correspondence at a coarse level with a region matching and bi-directional optical flow aggregation module. The usage of term "stroke" here align with the stroke-based methods, referring to every single lines or outlines in a sketch keyframe. Guided by these multi-level perspectives, a multi-stream U-Transformer architecture is further devised to produce the intermediate sketch frames. It consists of two attentionbased building blocks: convolution and self-attention block (CSB) and the convolution and cross-attention block (CCB) to leverage the diverse multi-level insights for producing precise inbetween sketch patterns. To facilitate the research on hand-drawn animation sketch inbetweening, we constructed a large-scale sketch triplet dataset: STD-12K, from 30 sketch animation series over 25 hours with various artistic styles. Comprehensive experiments demonstrate that our SAIN clearly outperforms the state-of-the-arts for animation sketch interpolation.Overall, the key contributions of this study are as follows:• A novel deep learning architecture, SAIN, for sketch interpolation by effectively formulating sparse sketch patterns with sketch-aware multi-level guidance. • A novel self / cross-attention based multi-stream U-Transformer design with the multi-level guidance. • A large-scale sketch triplet dataset with various artistic styles constructed for the research community. 
related work: 
 Frame interpolation aims to synthesise the intermediate frames between a pair of given frames. It can be categorised into three major approaches regarding frame contents: sketch interpolation [11, 17, 32, 35, 37] , video interpolation [2, 16, 18, 20, 21] , and animation interpolation [25, 30] .Sketch interpolation is to produce raw animated sketch frames to streamline the 3-stage process of 2D sketch animation. Generally, existing studies can be categorised into stroke-based [11, 35, 37] and image-based approaches [17, 32] . Stroke-based methods for hand-drawn frames generally start with stroke vectorisation, and then perform stroke deformation [35] or construct specific structure units with vertices [37] . Recently, the transformer-based methods Sketchformer [22] and AnimeInbet [29] were introduced, capitalizing on the success of this architectural approach in various computer vision tasks. However, the preliminary processing heavily relies on extra software or techniques, and human animators often need to fine-tune the results of vectorization to ensure a smooth and accurate portrayal of intended movements. This introduces additional complexity, and substandard vectorization quality can adversely affect the inbetweening process, which makes stroke-based methods challenging to apply into the animation workflow. While interactive matching algorithms have been investigated for this process in [37] , scalability issues become prominent when the number of strokes in frames increases with subpar quality of hand-drawns.Image-based methods were initially studied in [32] , where an as-rigid-as image registration and an interpolation scheme were introduced to bypass the vectorization phase inherent in strokebased methods, demonstrating its potential in dealing with intricate stroke patterns. Recently, optical flow of sketches has been adopted to characterize the motions of characters and objects within an animation. In LDFI [17] , a distance transform mechanism was adopted, which engages the intensity gradients of the sketches with an enhances optical flow estimation. However, this mechanism can compromise sketch details, particularly in complex scenarios where strokes undergo significant changes.Different from natural videos in the real-world, cartoon animations mainly consist of expressive strokes and colour pieces. They often contain various non-linear and exaggerated motions. SGCVI [13] allowed users to generate inbetween frames guided by one user-input sketch. AnimeInterp [30] introduced a segment-guided matching module to estimate the optical flow for different colour pieces separately and a recurrent prediction module to address non-linear motions. EISAI [3] was recently proposed with a forward-warping interpolation architecture SoftsplatLite and a distance transform module to improve the perceptual quality.Early video interpolation studies were based on the optical flows between two input frames to represent and formulate motion patterns, exemplified by methods using a bidirectional optical flow method, such as in [6] . With the success of deep learning techniques in diverse computer vision tasks, kernel-based methods integrated convolution neural networks (CNNs) [12, 19] for efficient motion estimation and generation. Recently, due to the great success of visual transformers [5, 14] , transformer-based methods with selfattentions have been studied for video interpolation, adaptively addressing long-range pixel dependencies. Self-attentions were utilized to formulate the representation of each input frame in [26] . Cross-attentions between the input frame pairs were further studied in [9] . Optical flows were introduced to the transformer modelling as well to assist the formulation of motion dynamics [16] . DQBC [40] followed the flow-based paradigm and integrated correlation modeling to enhance the flow estimation. Yet, it is challenging to obtain reliable flow information between two animation sketches.  
methodology: 
 As shown in Figure 2 , SAIN takes two consecutive sketch keyframes as input and outputs an interpolated intermediate frame. It involves region, stroke, and pixel-level guidance to accurately capture and recognize the sparse characteristics of sketch animations. Correspondingly, a multi-stream U-Transformer is devised with selfattention and cross-attention based building blocks to produce intermediate sketches in a multi-scale manner. In this section, we first explain the problem formulation, then the details of SAIN's key components. To clarify and avoid ambiguity, it should be noted that the term 𝑠𝑡𝑟𝑜𝑘𝑒 refers to each individual pen or brush movement that contributes to the creation of the complete sketch keyframes during drawing.Denote two consecutive animation sketch keyframes as I 0 , I 1 ∈ R 𝐻 ×𝑊 ×𝐶 , where 𝐻 , 𝑊 , and 𝐶 denote their height, width and the number of channels, respectively. Animation sketch interpolation takes the two sketch keyframes to estimate an interpolated sketch frame Ît ∈ R 𝐻 ×𝑊 ×𝐶 for the ground truth I t ∈ R 𝐻 ×𝑊 ×𝐶 , (0 < 𝑡 < 1), where t = 0.5 for an intermediate frame between the keyframes by following the existing practices in the literature [16, 17] .To capture pixel-level motion dynamics, optical flows are estimated between the keyframes I 0 and I 1 . This allows us to output refined sketch keyframes, denoted as I 0 and I 1 , that incorporate these motion dynamics. Pixel-level dynamics are good at characterizing the patterns of all pixels, and they can also maintain the patterns that might be missed in the subsequent sketch-and region-level pattern formulation. Specifically, given a target timestamp t, an optical flow estimator [16] predicts the bi-directional flows: O t →0 and O t →1 . Next, the refined keyframes with pixel-level motion dynamics can be obtained as:EQUATIONwhere W is an image warping function [7] to fuse the two inputs with a pre-defined sampling strategy.To characterise sketch-based motion patterns for interpolation, stroke-level correspondence is formulated between sketch keyframes. Specifically, a stroke keypoint matching and tracking mechanism is devised for this purpose, which assists in a cross-frame stroke understanding.Point-wise matching aims to produce a set of matched salient points between the input strokes in 𝐼 0 and 𝐼 1 , encompassing a salient point identification step for individual keyframes and a salient point matching step between the paired keyframes. First, given a sketch frame, the stroke salient points can be identified with their feature descriptors that characterise their local point-wise patterns, which can be formulated by algorithms such as SuperPoint [4] . We denote the identified salient points as p 𝑖 = (𝑥 𝑖 , 𝑦 𝑖 , 𝑐 𝑖 ) and their visual descriptor as d 𝑖 ∈ R 𝐷 𝑝 for the 𝑖 th detected stroke point. In detail, 𝑥 𝑖 and 𝑦 𝑖 are the coordinates of the 𝑖 th salient point, and 𝑐 𝑖 indicates the detection confidence. To this end, given the two keyframes, the stroke salient point detection finds 𝑁 𝐼 0 𝑠 and 𝑁 𝐼 1 𝑠 points with their local features as: p 𝐼 0 𝑖 and d 𝐼 0 𝑖 , 𝑖 = 1, ..., 𝑁 𝐼 0 𝑠 for I 0 , and p 𝐼 1 𝑗 and d 𝐼 1 𝑗 , 𝑗 = 1, ..., 𝑁 𝐼 1 𝑠 for I 1 , respectively. Salient point matching further establishes the point correspondence between paired keyframes in line with their feature descriptors and obtains a set of point pairs with confidence scores. Mathematically, we have their confidence as:EQUATIONwhere U is a function to evaluate the confidence with the point coordinates and descriptors. A SuperGlue [24] method is adopted for this purpose. As shown in Figure 2 , colorful line connections indicate the matched point pairs with high confidence scores over a threshold 𝜃 , where their colours indicate the magnitude of the confidence scores: red for a higher score and blue for a lower score. Point-wise tracking. For a brief interval between I 0 and I 1 , the movements are assumed to follow a linear path with respect to a temporal indicator 𝑡, 𝑡 ∈ [0, 1]. Mathematically, consider a matched pair of salient points, such as the 𝑖 th keypoint in I 0 and the 𝑗 th keypoint in I 1 ; we have:EQUATIONwhere p 𝐼 𝑡 𝑖 𝑗 is an estimation of the intermediate trace of the stroke salient point at time 𝑡.To this end, the traces of these salient points at time 𝑡 can be conceptualized as a 2D frame, characterized by their coordinates. We represent the stroke-level correspondence using this trace, denoted as P 𝑡 ∈ R 𝐻 ×𝑊 ×𝐶 , 𝑡 ∈ [0, 1]. This trace shares the same dimension as I 0 and I 1 .Sketch frames generally contain clear outline strokes and enclosed areas. To leverage this regional nature for interpolation, regioncorrespondence is constructed between two sketch keyframes. Specifically, regions can be identified as segmentation maps using methods such as the trapped-ball algorithm. For I 0 and I 1 , 𝑁 I 0 𝑟 and 𝑁 I 1 𝑟 regions are identified, respectively. Pre-trained CNN features for these maps can be formulated, allowing the pixel-based features within a region to be pooled as a 𝐷 𝑟 -dimensional vector, thereby characterizing each region. Given the coordinates and features of these regions, a match can be established between the regions across keyframes akin to stroke correspondence. For a region pair (𝑖, 𝑗), where 𝑖 indicates the 𝑖 th region in I 0 and 𝑗 indicates the 𝑗 th region in I 1 , bi-directional optical flows can be estimated based on their features as f t →1 (𝑖, 𝑗) and f t →0 ( 𝑗, 𝑖). By summing up all regional optical flows, we have F t →1 = (𝑖,𝑗 ) f t →1 (𝑖, 𝑗) and F t →0 = ( 𝑗,𝑖 ) f t →0 ( 𝑗, 𝑖). To this end, the keyframes are refined with region-level correspondence information as follows:EQUATIONwhere the W is an image warping function.A multi-stream U-Transformer is devised to characterise the inbetweening patterns by jointly considering region, stroke and pixellevel dynamics. These streams are based on two building blocks: convolution and self-attention block (CSB) and convolution and cross-attention block (CCB). CSB U-Transformer stream. This stream fully adopts the multilevel dynamics for the motion patterns between I 0 and I 1 . Specifically, a concatenation is conducted on patterns regarding pixellevel ( I 0 , I 1 ), stroke-level (P 𝑡 ) and region-level ( I 0 , I 1 ), followed by a number of convolution layers to obtain a coarse-level intermediate sketch representation X coarse . To further formulate the inbetweening patterns from a fine-level perspective, CSB with self-attentions [34] is introduced to construct a U-Net [23] like a stream with an encoder-decoder architecture.The encoder consists of a series of CSBs. Specifically, a CSB consists of a convolution layer for modelling local sketch patterns and a multi-head self-attention for a global modelling purpose. In pursuit of an overall encoder pyramid structure of 𝑆 scales, a downsampling operator is introduced with CSB to formulate a feature map at its corresponding scale. For the 𝑠 th CSB, which is for the 𝑠 th scale, 𝑠 = 0, ..., 𝑆 -1, its output feature map is obtained as:EQUATIONwhereX CSB 𝑠+1 ∈ R 𝐻 CSB 𝑠+1 ×𝑊 CSB 𝑠+1 ×𝐶 CSB 𝑠+1 .Particularly, the first CSB takes X CSB 0 = X coarse as its input. In detail, X CSB 𝑠 is first with a convolution layer for local modelling. Note that for notation simplicity we keep using X CSB 𝑠 as the convolution-filtered results for the following discussion. Next,X CSB 𝑠 is divided into 𝐾 CSB 𝑠 = 𝐻 CSB 𝑠 𝑊 CSB 𝑠 /𝑀 2𝑠 sub-patches of size 𝑀 𝑠 × 𝑀 𝑠 following the general practice of a visual transformer [15] . By treating the pixel-wise values within the 𝑘 th patch as a representation vectorx CSB 𝑠,𝑘 ∈ R 𝑀 2 𝑠 ×𝐶 CSB 𝑠 , X CSB𝑠 can be viewed in a matrix form, whereX CSB 𝑠 = [x CSB 𝑠,1 ⊤ , ..., x CSB 𝑠,𝐾 CSB 𝑠 ⊤ ].Then, the matrices of Key, Query, and Value in a self-attention can be computed to obtain a frame-level sketch understanding:EQUATIONwhere W indicates a matrix with learnable parameters for a linear projection. Next, the attention can be computed as:EQUATIONwhere 𝑑 CSB 𝑠 is the dimension of queries, keys and values. Specifically, we denote X fine = X CSB 𝑆 as a fine-level feature map. Note that this encoder structure, along with its CSB blocks, can work seamlessly with multi-head self-attentions, which enables the extension of frame-level sketch patterns from multiple perspectives. Furthermore, the structure is compatible with the Swin-based window strategies used during patch construction, keeping efficiency in consideration. Finally, the decoder with a number of deconvolution layers upsamples X fine as a synthetic CSB frame feature map ÎCSB t . CCB U-Transformer stream. In the CSB stream, multi-level patterns contribute equally to the formulation of the feature map.Leveraging the stroke and region-based characteristics of sketch animes, CCB U-Transformer streams with an encoder-decoder structure are devised to provide diversified modelling perspectives for sketch interpolation.For its encoder, similar to the CSB stream, a series of CCBs are adopted for different pyramid scales 𝑠 = 0, ..., 𝑆 -1. For the 𝑠 th CCB, a feature map is formulated as:EQUATIONwhereX CCB 𝑠 , Y CCB 𝑠 ∈ R 𝐻 CCB 𝑠 ×𝑊 CCB 𝑠 ×𝐶 CCB. Specifically, stroke-level patterns P 𝑡 are concatenated and downsampled with learnable convolutions as Y CCBfor key and value computations in the crossattention. In terms of query, pixel-level dynamics and region-level correspondence are utilized. Since they are bi-directional, dual CCB U-Transformer streams are introduced for concatenated query features I 0 and I 0 , or I 1 and I 1 . In particular, the first CCB takes F CCB 0 = 𝐶𝑜𝑛𝑣 ( I 0 , I 0 ) or 𝐶𝑜𝑛𝑣 ( I 1 , I 1 ) as its inputs, where 𝐶𝑜𝑛𝑣 is a function for concatenation and downsampling, (I 0 , I 0 ) and ( I 1 , I 1 ) can be viewed as the forward and backward streams.Finally, decoders produce synthetic intermediate feature maps: ÎCCB t,0 or ÎCCB t,1 are obtained from the two streams regarding the inputs X CCB 0 = I 0 and X CCB 0 = I 1 , respectively. Multi-stream fusion. Upon obtaining the intermediate feature maps ÎCSB t , ÎCCB t,0 and ÎCCB t,1 from the CSB and CCB streams, a fusion mechanism then yields the final estimation Ît of the interpolated frame I t , by which all feature maps are concatenated and go through a series of convolutions.We denote the computations of the proposed method as a function 𝑓 with learnable weights Θ. To obtain Θ, ℓ 1 reconstruction based loss is adopted to optimise pixel-wise difference between the ground truth sketch frame I 𝑡 and the interpolated frame Î𝑡 . Mathematically, we have:EQUATIONTo further improve the synthesized details, we apply a perceptual LPIPS loss [39] , denoted as L lpips . Jointly, the proposed model is optimized as:EQUATION) 
experiments & discussions 4.1 dataset: 
 Due to the lack of publicly available datasets for animation sketch interpolation, by following the protocols of existing video interpolation datasets such as Vimoe-90K [36] and UCF101 [31] , we constructed an animation sketch dataset based on ATD-12K [30] , namely Sketch Triplet Dataset-12K (STD-12K), for evaluation and facilitating the research on this topic. STD-12K is a large-scale sketch triplet dataset extracted from 30 animation movies with extensive artistic styles. To convert an animation frame to a sketch frame, Sketch Keras was first used to detect and extract the contours in a frame and rough strokes can be obtained. Next, a sketch simplification procedure [27, 28] was introduced to remove blurry and trivial strokes, which also refined the basic and necessary sketch lines. Figure 3 depicts the key steps to construct this dataset.Network architecture. For P 𝑡 in stroke matching and tracking, we specified a stroke correspondence sequence with 𝑡 = 0.5. An appropriate setting to use these temporal information would have impact on the interpolation accuracy as indicated in the experiments. For the optical flow computations, we first predicted the coarse flows using convolutional flow prediction network and then refined the coarse flows in a coarse-to-fine manner following an existing practice as in VFIformer [16] . For the CSB component, a swin-based strategy was adopted with a window size of 8 × 8. The number of channels in its convolution layer was set to 24. The CCB component was with the same setting as CSB. Each U-Net like transformer stream contained 3 CSBs or CCBs.Training details. The proposed method was trained using an AdamMax optimizer [10] with 𝛽 1 = 0.9 and 𝛽 2 = 0.999. The weights of loss terms were set to 𝜆 1 = 70 and 𝜆 lpips = 30. The training batch size was set to 4. The SAIN was trained for 50 epochs with a learning rate that initially was set as 2𝑒 -4 and a weight decaying factor was set to 1𝑒 -4 . The sketch frames were resized and cropped into a resolution of 384 × 192, and they were also augmented with a random flipping operator. It took approximately 72 hours on an NVIDIA A6000 GPU for the training procedure.Evaluation metrics. For quantitative evaluation, we adopt commonly used visual quality assessment metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) scores, Interpolation Error (IE) [1] which measures pixel-wise difference between the interpolated and ground-truth sketches, and Chamfer Distance (CD) which measures the dissimilarity between two sets of points. For readability, IE is scaled by 1𝑒2 and CD by 1𝑒4. Methods for comparisons. SAIN is compared with the recent state-of-the-art methods, encompassing stroke-based sketch frame interpolation method AnimeInbet [29] , Sketchformer [22] and imagebased LDFI [17] , animation-based methods SGCVI [13] and EISAI [3] , and video-based methods: Super SloMo [8] , AdaCof [12] , Soft-Splat [18] , RIFE [7] , VFIT [26] , VFIformer [16] and DQBC [40] . All the SOTA models were retrained on our proposed dataset, STD-12K, with the exception of AnimeInbet, for which we utilized the authors' pre-trained model. The decision to use the pre-trained AnimeInbet model stemmed from the challenges we encountered in applying the authors' recommended method for vectorizing our sparse images. This process spans over a month for preprocessing our training dataset and it creates unsatisfactory quality sketches that are not suitable for further training. Quantitative evaluation. As shown in Table 1 , our SAIN consistently outperforms the other methods with PSNR 20.32, SSIM 0.8727, IE 10.09 and CD 1.54. SAIN successfully addresses the sparsity nature of sketch animations, as evidenced by the lowest CD score which indicates a high degree of similarity between the interpolated sketch and the ground truth, given the sketch as a set of points. Video-based methods generally underperform when compared to the animation-based approach -EISAI. The only exception is VFIformer, which slightly surpasses EISAI in terms of the IE metric. This underscores the challenges that video-based methods face when dealing with the sparse patterns intrinsic to animations. When comparing our proposed SAIN to EISAI, it becomes evident that proper sketch-based mechanisms are essential, especially given that sketch-based animations often lack color and detailed texture structures. However, while AdaCoF performs well in terms of CD, its PSNR is poor, suggesting that it prioritizes rough sparse structures and ignores detailed patterns. Other video-based methods perform even worse in terms of CD metrics. EISAI, an animationbased method, achieves comparable PSNR and SSIM scores, but still struggles with sparsity, particularly without texture and color information. Finally, when compared to the sketch-based LDFI method, its intensity gradient mechanism results in inaccurate results, especially for complex scenarios with dramatic changes of strokes. Qualitative evaluation. Figure 4 illustrates interpolation examples from simple to complex scenarios for the qualitative comparisons among different methods. The frames with a black border are full sketch frames, and we zoom in a specified region within a red window to observe the detailed interpolated patterns. Overall, it can be observed that SAIN is capable to generate high-quality inbetweens, and the results produced by other methods generally have different type of artifacts such as blurriness and distortions. The first example is with the simplest strokes. The results produced by LDFI and DQBC missed many strokes due to the limitation in exploring sketch correspondence for alignments. While EISAI and VFIformer achieve improved performance, the issue of blurriness persists due to the lack of texture and color reference. With the increasing sketch complexity, these artifacts become more significant and the contents tend to be unrecognizable (e.g., the 3 th example with DQBC) and distortion (e.g., suspecting angry face with EI-SAI for the 3 th example). Note that in the zoom in region, AdaCof and LDFI failed to output contiguous strokes since insufficient correspondence information was extracted which result in missing Pixel-wise dynamics. By removing the pixel-wise dynamics, the CCB and CSB transformer streams take region-level correspondence I 0 and I 1 , and stroke-level correspondence P ′ 𝑡 as inputs. As shown in Table 3 , the absence of pixel-level motion information resulted in a deteriorated performance. Moreover, as shown in Figure 5 , the interpolation examples exhibits blurring results, potentially due to the uncertain direction of pixel-level motion. Stroke-level correspondence. The stroke-level correspondence P 𝑡 was removed from SAIN by excluding the CCB based transformer streams and P 𝑡 in the CSB based transformer stream. Only the four refined sketch frames I 0 , I 1 , I 0 and I 1 were adopted and fused 5 , the intricate details of the clown's face is difficult to discern when CCB is not utilised. Moreover, CCB facilitates a robust learning with stroke correspondence, whilst an improper stroke guidance usage may result in inferior interpolation results for some scenarios.Convolution self-attention block. When the CSB block is omitted, the quantitative results, as depicted in Table 5 , demonstrate inferior performance compared to the full SAIN. In particular, in the first example (inside the princess's hair) illustrated in Figure 5 , blurriness occured in the princess's hair when interpolation without CSB.Region-level correspondence. Similar to the removal of pixelwise dynamics, by removing the region-level correspondence, we instead take the refined outputs from pixel-wise dynamics module I 0 and I 1 to explore another situation not fully utilizing the correspondence. SAIN without region correspondence produces some blurriness within the closed boundaries as shown in Figure 5 (row with red outlines), which demonstrates that region correspondence helps refine closed areas. Overall, the absence of region-level correspondence also leads to a decreased performance compared with the full SAIN.The stroke correspondence P 𝑡 is continuous in terms of 𝑡. An ideal temporal modelling strategy needs to incorporate sufficient information without causing issues due to redundant input patterns. We investigated a number of settings as shown in Table 3 . It can be observed P 𝑡 with 𝑡 ∈ { 1 2 } achieves the best performance. Conversely, providing more information may lead to an over-fitting issue, especially for the case with 𝑡 ∈ { 1 4 , 1 2 , 3 4 }. As expected, only with the guidance of the middle temporal point 𝑡 ∈ { 1 2 } works the worst among all settings. However, with three sampling slices, where 𝑡 ∈ { 1 4 , 1 2 , 3 4 }, the performance is worse than all two-slice based strategies, which suggest an over-fitting issue. For two-slice strategies, the most effective strategy is with the early motion patterns in 𝑡 = 1 4 , highlighting the importance of the initial states.Our proposed SAIN relies on the result of stroke and regional correspondence, which leads to limitations with the current correspondence mechanism. First, a linear stroke-level correspondence and a pre-defined temporal sampling strategy may result in less accurate interpolations. The future work should address this stroke-level correspondence scheme with a learnable manner based on the input sketches. Second, improvement in the scenarios of exaggerated motions is expected. In Figure 6 , it can be observed that the strokes change extremely between the frames. As a result, the stroke keypoints are often with less confidence or incorrect for the point-wise matching and downstream modelling. We present interpolation results for two frames featuring significant motions, as illustrated in Figure 7 , alongside comparisons with existing Sketch (LDFI), Animation (EISAI) and Video (Adacof) interpolation state-of-the-art methods. Notably, Adacof and LDFI were unable to accurately capture the images within such a dynamic scene, resulting in blank outputs. EISAI, on the other hand, compromised the detail of the foot, leading to distortion, while our method successfully preserved the overall structure of the input. 
conclusion: 
 In this paper, a novel deep learning method, namely, SAIN, is presented for animation sketch interpolation. Particularly, region-, stroke-, and pixel-level patterns are explored to take the sparse nature of sketch frames into account for interpolation. A multistream U-Transformer architecture is further devised to utilise the multi-level guidance with CSB and CCB. In order to evaluate our proposed method, a large-scale sketch dataset extracted from wild animation STD-12K was constructed for the first time. Comprehensive experiments clearly demonstrate the effectiveness of SAIN against the-state-of-the-art.  
 
 Appendix:  


Decision: This paper was reviewed by four experts in the field. The paper received postive reviews WA, BA, BA, BA. 
Reviewers like the new large-scale dataset, but raise the concerns of missing reference, novelty and bias of the dataset, etc.
After rebuttal, the reviewers come to a consensus that the concerns are well solved. 
Considering the reviewers’ concerns, we are happy to recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: The paper presents a new method - Sketch-Aware Interpolation Network (SAIN) for high-quality sketch animation inbetweening, which is an important yet challenging task with a wide spectrum of applications. A large-scale dataset has also been developed to be used in this work and further relevant studies. Extensive experiment results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: Overall, the paper is well organised and easy to follow, and has made solid technical contribution to the sketch animation community. The newly developed dataset will also facilitate subsequent studies in this area.
limitations: limitations: The authors have provided a group of animations obtained with different methods in the supplementary demo video. They are encouraged to provide more examples in the demo video, for instance, all the instances that appear in the manuscript, and this will be helpful in evaluating the overall visual quality of the proposed method.

In addition, it would be interesting to see a comparison between different methods in handling sketches with varied details. For example, detailed sketches to highly abstractive sketches.
paper_summary: paper_summary: The authors propose a hand-drawn 2D in-betweening pipeline that utilizes motion dynamics, stroke-level correspondence, and region-level correspondence as inputs. Although more details on how these inputs are calculated are needed, the contribution seems adequate with the introduction of a new dataset.
strength_weakness: strength_weakness: 1. The challenges of limited texture details and exaggerated changes between frames are well-identified, and the model design effectively addresses these challenges by constructing three correspondences and dynamics. Although it is unclear the detailed calculation of those correspondences and whether these correspondences are proposed for the first time, building such dynamics is pretty hard for sketch videos and does well represent the motion inside sketch animation.
2. A large-scale dataset, STD-12k, is proposed and claimed to be publicly available. 
3. The Sketch-Aware Interpolation model fuses three correspondences and dynamics and appears novel and promising with detailed ablation studies.
limitations: limitations: 1. Can this model be extended to handle freestyle, poorly-drawn sketches? It seems that the sketches used are derived from edge detection.
2. More details are needed on how optical flow is estimated between frames. It is unclear what model is involved or how the optical flow is calculated.
3. The abstract briefly mentions color details (#14), but there is no further clarification on what these color details are and how they relate to the grayscale sketches.
4. Some typos need to be carefully checked like inbewteening in #20
paper_summary: paper_summary: The work introduces a new method called SAIN for the sketch interpolation task. It features a novel design that integrates multi-level sketch-related guidance with U-Transformers. Additionally, a new large-scale dataset is proposed.
strength_weakness: strength_weakness: The paper introduces a new dataset that could potentially contribute to the development of the field. Additionally, the work achieves excellent results on the proposed dataset and provides a detailed experimental analysis. The comparison of this work with other baselines highlights its strengths and weaknesses, offering valuable insights for future research.
limitations: limitations: I apologize for any misunderstanding, but after carefully reading the paper, I still have the following questions:

1. I do not clearly understand the necessity of proposing this new dataset. Methods such as AnimeInbet also provide datasets, so why not compare your work on these existing datasets? Additionally, I noticed that the performance of the AnimeInbet method in Table 1 is quite poor, but there is a lack of analysis regarding this.

2. I do not fully grasp the advantages of your proposed dataset, nor the relationship between your dataset and existing related datasets. Is there an evaluation of your method's performance on other datasets?

I might have overlooked certain parts of the paper that address these points. If you could clarify these aspects, I would greatly appreciate it and improve the score.
paper_summary: paper_summary: This work proposed a sketch-aware interpolation network for animation sketch inbetweening. Given the sketching keyframes, the approach firstly computes multi-level correspondences, i.e. pixel-level dynamics, stroke-level correspondence and region-level correspondence. Then guided by these correspondences, a multi-stream U-transformer is devised to generate the inbetweening frames with CSB and CCB. Moreover, this work constructed a sketch dataset termed STD-12K which comprises 30 sketch animation series.
strength_weakness: strength_weakness: 1.	This work investigates pixel, stroke, and region-level guidance to accurately comprehend the sparse characteristics of sketch animations. 
2.	Extensive experiments have demonstrated the effectiveness of the proposed approach.
limitations: limitations: 1．	This work lacks an important ref [1], which also converts open-source movies to full scenes in 2D styles including 2D Animation frames and contour lines. It provides ground-truth correspondences at pixel-wise and region-wise levels, respectively, where the pixel-wise labels are dense optical flows of cartoon sequences, and the region-wise ones are the matching of segments in adjacent frames. Many evaluation metrics have also been provided. 

2．	Compared to reference [1], it is evident that this article lacks novelty. Firstly, all the modules proposed in this work, including pixel-level dynamics, stroke-level and region-level correspondences, transformer architecture, and the dataset, have been exploited in the field. Secondly, reference [1] provided ground-truth correspondences at pixel-wise and region-wise levels, along with corresponding evaluation metrics like EPE, occ, line, and flat, which are not included in this work. Thirdly, this work fails to evaluate the accuracy of correspondences at multiple levels. Additionally, the evaluation results in Tables 1 and 2 do not reach the SOTA compared to reference [1].

3.	Some limitations have been discussed in the limitations section of this article, such as exaggerated and large amplitude movements, and inaccurate stroke-level correspondence, which limit the practicality of this method.

4.	The proposed dataset STD-12K is extracted from 30 animation movies. Apparently, there is a significant bias between these data and hand-drawn sketches. Considering that the first sentence of this article emphasizes hand drawing, I believe the author needs to discuss hand-drawn sketches more, such as noise in keyframes and semantic confusion.

Ref:
[1] AnimeRun: 2D Animation Visual Correspondence from Open Source 3D Movies. Siyao, Li and Li, Yuhang and Li, Bo and Dong, Chao and Liu, Ziwei and Loy, Chen Change,  NeurIPS 2022.