Towards Photorealistic Video Colorization via Gated Color-Guided Image Diffusion Models

Authors: jiaxing  li
hongbo  zhao
yijun  wang
jianxin  lin
 2024 towards
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:25:16

Abstract: Video colorization poses challenging tasks, necessitating structural stability, continuity, and details control in the colors produced.  In this paper, based on a pretrained text-to-image model, we introduce the $\textbf{Gated Color Guidance}$ module ($\textbf{GCG}$), enabling the model to adaptively perform color propagation or generation according to the structural differences between reference and grayscale frames. Based on this multifunctionality, we propose a novel two-stage coloring strategy. In the first stage, under reference-mask condition, the model autonomously and jointly colors input keyframes in a one-to-many color domain mapping, while temporal coherence constraints are emphasized by modifying the attention mechanism. In the second stage, under reference-guided condition, the model effectively captures the colors of matching structures in the reference, and we further introduce $\textbf{Sliding Reference Grid}$ strategy ($\textbf{SRG}$) to merge and extract the color features from multiple frames, providing more stable coloring for the grayscale frames. Through this pipeline, we can achieve high-quality and stable video coloring while maintaining the accuracy of detailed colors. Additionally, the two-stage strategy is flexible and detachable, allowing users to adjust the number of selected reference frames to balance coloring quality and efficiency. Extensive experiments demonstrate that our method significantly outperforms previous state-of-the-art models in both qualitative comparison and quantitative measurement.

Main paper: 
 introduction: 
 In multimedia processing, video colorization is essential for both aesthetics and cultural heritage restoration. The task involves enriching grayscale videos with true-to-life colors, necessitating precision and temporal continuity, while maintaining vividness of colors and stable detail control.Previous methods of video colorization can broadly be categorized into three types: The first solution involves coloring each frame independently and then applying temporal smoothing [5, 30, 32, 38] . This method is heavily reliant on the original coloring results, and the smoothing tends to darken the footage, reducing its color richness. The second solution relies on single frame colorization and color propagation [10, 19, 40, 56] . This approach is prone to the accumulation of errors, causing the hues in later frames to gradually deviate from the original color domain. The third solution encompasses fully-automatic video colorization [29, 31, 48, 70] ; however, the characteristic of this category is a lack of detailed processing. It struggles with samples that include complex movements, often resulting in color artifacts, and generally fails to achieve a rich color palette.Our work adapts the diffusion model to the task of video colorization, aiming to address the aforementioned issues. One consideration is to adopt text-to-video (T2V) generation methods [14, 15, 46] using extensive text-video datasets for high-quality colorization. However, these methods are cost-prohibitive and unsuitable for widespread content creation. Thus, we prefer fine-tuning on pretrained text-to-image (T2I) diffusion models. [55, 67] provide valuable templates, expanding self-attention to encompass all frames, allowing a single attention query to access key-value information from all frames and share global information for overall temporal consistency. But this global query operation increases the computational cost during inference, potentially lengthening the colorization time, which could reduce competitiveness in video colorization tasks that have strong structural cues. Based on these analyses, we aim to fully leverage the pairing information of color and structure in the video colorization task. We intend to maintain temporal consistency not solely through global queries but by aggregating information across multiple frames using different strategies. Moreover, previous work reminds us that we should pay more attention to coloring detailed areas and issues such as color leakage.We propose a new video colorization pipeline, first jointly autocolorizing selected keyframes and then mapping remaining frames to the corresponding color domains of these keyframes, culminating in full video coloring. This requires our model to adaptively perform color propagation or generation based on the differences between the input reference frames and grayscale frames. Therefore, we introduce Gated Color Guidance module (GCG) into the pre-trained T2I diffusion model, using gating mechanisms to achieve high-quality colorization under different conditions. In detail, under reference-mask condition, it autonomously colorizes input grayscale frames in a one-to-many color domain mapping; under reference-guided condition, it effectively captures and applies the color palette of the reference to grayscale frames in a one-to-one mapping. Moreover, when the reference is available and valid, this gated attention selects structure-color matching information from the reference, filtering out mismatched color information to prevent redundant feature aggregation and stabilize detail colorization.Specifically, our two-stage coloring strategy involves: in the first stage, key frames are jointly colorized under reference-mask conditions. By employing an extended attention mechanism, these key frames share global information amongst themselves to ensure consistency in colorization; in the second stage, we further explore full video coloring. Differing from the first stage, where each attention operation was expanded to a global query, trading time for color consistency, we propose a data augmentation method specifically designed for video colorization tasks, termed the Sliding Reference Grid (SRG). Here, unlike [23] , which directly edits the combined grid, we use the grids from multiple video frames as a reference to guide the colorization of images, akin to a sliding window approach. This allows colorizing frames to obtain information from different frames at various time steps, achieving a more stable colorization effect. During fine-tuning, we provide the model with three kinds of inputs: reference-mask, single reference image, and reference grid to foster its capacity for generalization, and through our method, the model is capable of generating stable and highquality results, as shown in Figure 1 . Using common datasets and evaluation metrics, we provide a comprehensive evaluation of the trained model. Extensive experiments verify the superiority of our model in color accuracy, richness, and temporal consistency for video colorization tasks.Our contributions are summarized as follows:• We present a novel framework that addresses the challenge of video coloring, decomposing the colorization steps into Keyframes Joint Coloring and Full Video Coloring, which achieve flexible, stable, and high-quality video coloring. • We propose the Gated Color Guidance module (GCG), which adaptively performs color propagation or generation by assessing the gap between the reference and input grayscale frames, thus underpinning our two-stage coloring strategy and ensuring accurate colorization and stable detail control. • We introduce the Sliding Reference Grid strategy (SRG), utilizing the diffusion model's emphasis on spatial relationships to enable single grayscale frames to capture information from more frames, producing stable outcomes. 
related work 2.1 image colorization: 
 Early image coloring networks were relatively simple and direct in structure. Similar to other CNN tasks, Zhang et al. [65] proposed an end-to-end network for learning the mapping from grayscale images to quantized chromaticity distributions, thus achieving automatic coloring. With the development of Generative Adversarial Networks (GANs) [9] , combining the advantages of GANs with learning semantic category distributions has been used to accomplish coloring tasks [27, 50] . However, these methods still have shortcomings in terms of semantic consistency and color richness.To address these challenges, leveraging Transformer's ability for remote context extraction [20, 53] , models have been developed to predict a diverse range of color tokens in an autoregressive manner. Nevertheless, for coloring tasks, users typically desire the ability to color images according to their preferences. Approaches such as [45] and [8] utilized deep adversarial image synthesis architectures to enable users to provide coloring conditions in the form of scribbles. Yun et al. [62] utilized the global receptive field of Transformers to propagate the scribbles provided by users to relevant regions. Coloring methods based on reference images transfer color information from reference images to grayscale images using the color histogram as a prior [61] or through global [51] or local similarity [6] . Additionally, Manjunatha et al. [39] , L-CoDe [54] and L-CoIns [7] applied language models to coloring network tasks to achieve text-controlled coloring.Video colorization, distinct from image colorization, requires further consideration of color continuity between frames and consistency of colors with the same semantics. Some post-processing methods [5, 30, 32, 38] are based on pre-trained image colorization models, where each frame is independently edited and then further corrected for temporal discontinuity through post-processing. Despite reducing flickering, these methods often produce frames with insufficient color continuity and faded colors. Another category of methods [10, 19, 40, 56] involves coloring the first frame individually as an example and then sequentially coloring subsequent frames. While these methods offers some degree of color stability, failures in coloring specific frames can lead to a decrease in the propagation effectiveness of colors. Meanwhile, errors may accumulate over time as the frame position shifts, resulting in a gradual deviation of colors from the authentic domain. Fully-automatic methods [29, 31, 48, 70] integrate image colorization and temporal smoothing together. They directly map grayscale frames to their color embeddings via deep neural networks while considering frame continuity. However, these methods struggle to generate color-rich results, and the required time cost is excessively high. Some other methods [4, 18, 57] rely on additional examples provided by the user for coloring, including propagating user scribbles [33, 60, 69] or attaching colors from a reference image to the rest of the frames [59, 63] . However, these methods heavily depend on the quality of the given examples and their match with the video to be colored, and they are prone to washing out colors in some details.Methods based on Diffusion Models (DMs) are rapidly advancing, and their outstanding generative capabilities have been applied in various domains, such as image generation [35, 41, 43] and image editing [3, 12, 24] . Particularly, with the introduction of Latent Diffusion Models (LDM) [44] , a method of controlling image and video colorization through conditional injection into the diffusion process has emerged. Specifically, ControlNet [64] enables various conditions to participate in the denoising process of the diffusion model. It integrates more controllable input conditions into the text-to-image synthesis process using a pre-trained latent diffusion model, significantly enhancing the functionality of the diffusion model framework. Building upon this work, the CtrlColor [36] model proposes highly controllable colorization by leveraging combinations of multi-modal conditions. Liu et al. [37] introduced a dedicated coloring assistance module for DM, which has also been extended to the domain of video colorization. Additionally, a significant amount of notable diffusion model work focuses on video editing [11, 26, 55, 67] , which can often be applicable to video colorization. Inspiring works such as Tune-A-Video [55] extend the two-dimensional latent diffusion model (LDM) to three dimensions and introduce spatio-temporal attention. ControlVideo [67] extends full cross-frame attention on the original ControlNet for joint editing of all frames, followed by interleaving frames to ensure result stability. However, considering the mutual information between each frame in an n-frame video, we incur computational costs proportional to 𝑂 (𝑛 2 ), where some information is redundant.In contrast, our work first maps keyframes to the same color distribution through joint editing, and then uses these keyframes as references to colorize the remaining grayscale frames through a color distribution control module. By selecting the number of keyframes as a hyperparameter, we allow users to freely balance between precision and faster generation speed. 
method 3.1 overview: 
 Our work proposes a framework for stably colorizing grayscale video sequences 𝑉 = {𝐼 𝑖 } 𝑛 𝑖=0 , which produces high-quality videos with rich colors guided by text prompts. To balance coloring quality and computational efficiency, we employ a two-stage strategy. Initially, under the reference-mask condition, we conduct joint coloring on 𝑚 frames that are abundant in structural and color information. Subsequently, the remaining grayscale frames are mapped to align with the color domain of the key frames. We leverage the pretrained text-to-image (T2I) model and introduce the Gated Color Guidance Module (GCG) to effectively realize high-quality colorization under different conditions using a gating mechanism (Section 3.2). Subsequently, we will demonstrate the two stages of coloring: in Section 3.3, we outline the strategy for selecting keyframes and how to jointly colorize keyframes under reference-mask condition; in Section 3.4, we discuss how to colorize the full video with the keyframes. We introduce the Sliding Reference Grid as a data augmentation method that allows the frame to be colored to merge and extract color information from more frames without altering the network structure, resulting in more stable outcomes. The overall architecture of the proposed method is shown in Figure 2 .We propose the GCG module with the aspiration of achieving the following in supervised video colorization tasks: when reference frames 𝐼 𝑟𝑒 𝑓 are available, the model can extract effective and structurally matched color information from the reference, mapping the frame to be colored to the corresponding color domain, where the mapping is one-to-one; in the absence of reference or in the parts not involved by reference, the model can freely colorize in a variety of ways, where this mapping characteristic is one-to-many. This multifunctionality underpins our two-stage strategy during inference, and we hope that color propagation with reference will capture more details. Specifically, we designed the following modules, realizing these functions through fine-tuning on datasets.Color Guidance Attention. Firstly, we introduce the Color Guidance Attention(CGA) module to achieve the fundamental color propagation functionality, meaning that we aim to guide the frame to be colored 𝐼 𝑖 to obtain color features that match the colored reference frame 𝐼 𝑟𝑒 𝑓 . We define the representations of 𝐼 𝑖 , 𝐼 𝑟𝑒 𝑓 , and the grayscale reference 𝐼 𝑟𝑒 𝑓 𝑔 in the latent space as 𝑔 𝑖 , 𝑧 𝑟𝑒 𝑓 , 𝑔 𝑟𝑒 𝑓 . By modifying the traditional cross-attention mechanism [49] , we map the feature representations 𝑔 𝑖 , 𝑔 𝑟𝑒 𝑓 , 𝑧 𝑟𝑒 𝑓 to query 𝑄, key 𝐾, and value 𝑉 through linear transformations 𝑊 𝑞 , 𝑊 𝑘 , 𝑊 𝑣 , respectively, as follows:TextEQUATIONThen the result is given by:EQUATIONwhere √ 𝑑 is a scaling factor. Since 𝑔 𝑟𝑒 𝑓 and 𝑧 𝑟𝑒 𝑓 maintain strict structural alignment, and 𝑔 𝑖 and 𝑔 𝑟𝑒 𝑓 are both grayscale image features, this structure ensures the acquisition of basic color lowdimensional information that matches.Color Distribution Gated Attention. Building upon CGA, we further propose the Color Distribution Gated Attention (CDGA) as constraint on the aggregation degree of low-dimensional color information output by CGA with the original information. In CGA, despite the spatial relevance between 𝑔 𝑖 and 𝑔 𝑟𝑒 𝑓 , they are not perfectly aligned. The aggregation of features from unrelated areas can cause negative impacts. If the difference between 𝑔 𝑖 and 𝑔 𝑟𝑒 𝑓 increases, 𝑔 𝑖 through CGA query obtains more mismatched color information, leading to instability in the final generated effect (which will be discussed in detail in our ablation study in Section 4.5.1). Therefore, an additional control can be applied using a gating mechanism to smooth between 𝑔 𝑖 and the results queried by 𝑔 𝑖 through CGA, fading out mismatched color information to prevent redundant feature aggregation.The implementation of CDGA differs from CGA; we map 𝑔 𝑖 and 𝑔 𝑟𝑒 𝑓 onto the Q, K, and V of cross attention as follows:EQUATIONThen, we map the result range to [0,1], as shown in the following equation:EQUATIONwhere, 𝜎 denotes the activation function. And unlike the implementation of gated attention in references [1, 35] , here we use CDGA as a regulatory factor. Through the design of these two modules, our final aggregation result of low-dimensional color reference features is:EQUATIONIn this way, by regulating with the factor output by CDGA, we can effectively filter out mismatched structural features, preventing negative impacts from redundant color information aggregation. After obtaining an effective ColorHint, we concatenate it with the original grayscale feature 𝑔 𝑖 , serving as input to the UNet copy. In addition, under reference-mask condition, the reference is set as an all-zero mask, i.e., 𝑔 𝑟𝑒 𝑓 = 𝑧 𝑟𝑒 𝑓 = ∅, and the regulation by CDGA results in ColorHint ≈ 𝑔 𝑖 , thus coherently achieving the dual functionality mentioned earlier. Furthermore, for the UNet copy, we also utilize the CLIP image encoder to leverage the information of 𝑧 𝑟𝑒 𝑓 as a global condition [16, 68] , implementing cross-attention. Utilizing the shared feature space with the text encoder, it provides semantic features of the reference image, serving as a beneficial initialization to accelerate the entire network training process.This section considers how to uniformly colorize keyframes unconditionally. For a grayscale video 𝑉 = {𝐼 𝑖 } 𝑛 𝑖=0 with 𝑛 frames, we first need to consider how to select the keyframes. Our approach is straightforward: if we need to select a set of 𝑚 keyframes {𝐼 where 𝐷 (•, •) is a measure of dissimilarity between the video frames𝐼 𝑘𝑒𝑦 𝑖and 𝐼 𝑘𝑒𝑦 𝑗 . In our actual work, we take the first and the last frames and then select additional 𝑚 -2 frames to join the keyframe set.Next, we aim to colorize the obtained set of keyframes {𝐼} 𝑖 ∈k with reference-mask condition. However, in our proposed GCG module, the gate conditions for reference-mask lean towards oneto-many colorization. Therefore, independently colorizing each keyframe, i.e., 𝜖 ′ 𝜃 (𝑧 𝑘𝑒𝑦 𝑖,𝑡 , 𝑡, 𝜏, ∅) and 𝜖 ′ 𝜃 (𝑧 𝑘𝑒𝑦 𝑗,𝑡 , 𝑡, 𝜏, ∅), does not guarantee the consistency and stability of the overall color. To address this issue, following previous work [55, 67] , we extend the self-attention module to simultaneously process these keyframes, thereby allowing the set of keyframes to share global color. Specifically, if at time step 𝑡 during the inference stage, the noisy feature map 𝑧 𝑘𝑒𝑦 𝑖,𝑡 of 𝐼 𝑖 is mapped to the corresponding query 𝑄 𝑖 , key 𝐾 𝑖 , and value 𝑉 𝑖 upon entering the main Unet's self-attention, the original query result for the query 𝑄 𝑖 of frame 𝑖 is:EQUATIONWe extend the self-attention module by concatenating the keys and values of all frames, allowing 𝑄 𝑖 to query all frames in the keyframe set, resulting in the new attention output:EQUATIONwhere K and V are respectively the results of concatenating the elements within the sets {𝐾 𝑖 } 𝑖 ∈k and {𝑉 𝑖 } 𝑖 ∈k . In this way, each keyframe's query can aggregate color information from all keyframes, leading to a consistent overall color. It's found that the original attention only queries itself, with an average computation cost equivalent to 𝑜 (1) for a single frame query, while the computational cost of the extended attention mechanism is 𝑜 (𝑚 2 ). Theoretically, we could take all frames of the original video as keyframes, i.e., 𝑚 = 𝑛, but this would require longer computation time and more computational resources.After the keyframes have been colored {𝐼 𝑘𝑒𝑦 𝑖 } 𝑖 ∈k , we need to further color the remaining frames {𝐼 𝑘𝑒𝑦 𝑖 } 𝑖 ∈𝑈 [0,𝑛]-k , while considering how to maintain temporal coherence.3.4.1 Sliding Reference Grid. Continuing with the idea of Key joint coloring, we need the noisy feature map of the 𝑖-th frame at time step 𝑡 to query information from more frames. We aim to find a method with lower computational cost, instead of expanding the self-attention module, thereby trading time for color consistency. A characteristic of video coloring tasks is the strongly structured reference of grayscale frames, which to some extent allows us to combine images into a grid. Different from [1] where the combined grid replaces all inputs in the pipeline, here the existence of the grid acts as a form of data augmentation, combining only reference frames. Meanwhile, the main Unet backbone processes a single video frame at a time. This prevents the decrease in generation quality due to the combined grid. We call this strategy sliding reference grid. Specifically, we sample from keyframes and already colored video frames, and through a splicing method, they form a new reference image. Then we map its colored and grayscale frames to the inputs 𝑧 𝑟𝑒 𝑓 and 𝑔 𝑟𝑒 𝑓 of the GCG, guiding the to-be-colored grayscale image 𝑖 as follows:EQUATIONRegarding the specific sampling strategy, for the latent feature of the i-th frame at time 𝑡 in the denoising process, i.e., 𝑧 𝑖,𝑡 we aim to satisfy the following optimization condition:EQUATIONwhere 𝐷 (•, •) is defined as in Section 3.3, 𝑐 denotes the size of the reference grid, meaning that during the inference phase, each frame can reference information from 𝑐 frames. Based on this formula, in the early stages of the denoising process, we refer more to key frames to capture global color information, while in the later stages of denoising, we refer more to the neighboring frames of the frame being colorized, to maintain stability and coherence in details between frames.In order to ensure that our model accurately possesses the capability to adaptively perform color propagation or generation according to the structural differences between reference and grayscale frames, as well as control over detailed colors, we need to fine-tune the GCG module. During the training phase, we will randomly sample from three types of references: reference-mask, single reference image, and reference grid. The loss function for training the model is defined as the Mean Squared Error (MSE) between the predicted noise 𝜖 𝜃 and the actual noise 𝜖: where 𝑈 (1,𝑇 ) represents the uniform distribution over the set {1, . . . ,𝑇 }, and N (𝜇, Σ) denotes the multivariate Gaussian distribution with mean 𝜇 and covariance Σ. Moreover, to ensure that the colorized output maintains structural similarity with the original grayscale input image, we only predicted the 𝐴𝐵 channels from [22] . Employing a strategy akin to [36] , we first transform the decoded color image 𝑦 into LAB space to obtain 𝑦 𝐿𝐴𝐵 and then extract only its AB channel, denoted as 𝑦 𝐴𝐵 . Finally, we concatenate 𝑦 𝐴𝐵 with the 𝐿 channel of the input grayscale image to form the ultimate colorized result 𝑦 ′ . This approach effectively preserves the original structural features.EQUATION 
experiments 4.1 implementation details: 
 Our model is initialized with the public weights of Stable Diffusion 2.1 [44] , following the Adam optimizer [28] with a learning rate set to 5 × 10 -5 . We conducted training for 15 epochs with a batch size of 24. Training was performed at a resolution of 448 × 256 using NVIDIA A100 GPU. For the inference process, we accelerated using Denoising Diffusion Implicit Model (DDIM) [47] sampling, set-ting the sampling steps to 32, and setting the guidance scale in the classifier-free guidance approach to 9.0.During training, we randomly sample from reference-mask, single reference image, and reference grid as reference for colorization, i.e., 𝑧 𝑟𝑒 𝑓 , 𝑔 𝑟𝑒 𝑓 . The reference grid consists of a grid made up of four randomly selected images, including both grayscale and color frames. To ensure diversity in training data, we utilized the HSV model for image color data augmentation, ensuring a variety of colors. For model prompt data, we employed BLIP [34] for textto-image transformation to obtain prompts. To ensure the model produces colorized results with a variety of colors, there was a 20% probability that the input is under reference-mask conditions, while colored words were removed from the generated prompts.We employed the Large-scale Diverse Video 3.0 (LDV 3.0) [58] dataset and the DAVIS 2017 [42] dataset as benchmark datasets for training and testing purposes. The LDV 3.0 dataset comprises 365 high-quality videos, encompassing various content categories, motion types, and frame rates. From this, we selected 335 videos for training and reserved the final 30 videos for testing. Additionally, the DAVIS 2017 dataset is a high-quality, high-resolution densely annotated video segmentation dataset, comprising 90 training videos and 30 DAVIS-Test-Dev 2017 testing videos. All video frames were uniformly resized to 448×256 pixels. To obtain grayscale frames, we adopted the approach outlined in [21] , utilizing the 'cv2.cvtColor()' operation for conversion to grayscale frames.To evaluate video colorization effectiveness, we consider metrics related to the quality, realism, color vividness, and temporal consistency. PSNR and LPIPS [66] assess image quality and similarity to reality under low-level perception. MUSIQ [25] captures multi-scale structural and statistical regularities, aligning well with human perceptual judgments. CLIP-IQA [52] evaluates semantic distance between images and descriptions, aiding image quality assessment. Colorfulness metric measures color vividness. CDC [38] evaluates color distribution consistency and temporal consistency in videos. FID [13] assesses feature statistics similarity between generated and real images. To validate the effectiveness of our method, we will compare it with several state-of-the-art methods, including automatic colorization methods such as Deoldify [2] , TCVC [38] , VCGAN [70] , and exemplar-based methods that require a reference exemplar as guidance, such as DRemaster [17] , DExemplar [63] , and BiSTNet [59] .For the exemplar-based methods, we will choose the ground truth of the first frame as the reference exemplar. For our model, we also employ the first frame as a reference frame (initially utilizing the Keyframes Joint Coloring method described in Section 3.2, where a reference-mask image is input to generate four consecutive keyframes. Subsequently, these four keyframes are utilized as reference frames, and the strategy outlined in Section 3.3, Full Video Coloring, is applied to complete the full colorization of the grayscale video). Regarding the model prompt input, we similarly utilize BLIP to perform image-to-text transformation for prompt acquisition.We conducted both quantitative and qualitative evaluations on the synthetic LDV3.0 test datasets and DAVIS-Test-Dev 2017 test datasets, with the results presented in Figure 3 , and Figure 4 and Table 1 , respectively. In terms of quantitative comparisons, our method outperforms other state-of-the-art methods significantly. Specifically, our method achieves superior results in terms of LPIPS, MUSIQ, FID, and Colorful metrics, indicating that our colorization results are of higher quality and more in line with human perception, demonstrating exceptional performance. It is worth noting that our method is able to utilize the latent color information contained in the pre-trained diffusion models. As shown in Table 1 , our method has a clear advantage over all other methods in terms of the Colorfulness metric. Although our method ranks third in the CDC metric on the synthetic LDV 3.0 test set, this is primarily attributed to our outstanding performance in the Colorful metric, surpassing all other methods. Our results exhibit more vivid and rich colors, which may lead to a higher CDC score. In contrast, the TCVC method, which achieves the optimal CDC score, performs poorly in the Colorful metric, indicating subdued colors that may result in a lower CDC score. Our qualitative evaluation results in Figure 3 also support this observation. In addition, in the qualitative comparison depicted in Figure 3 , and Figure 4 , our method not only demonstrates advanced capabilities in terms of color coherence and visually pleasing colorization results compared to automatic colorization methods, but also exhibits high color coherence compared to video colorization methods with reference. As the time frames lengthen, scenes may exhibit color characteristics that the reference frames did not possess. In such cases, our method benefits from Color Distribution Gated Attention to achieve consistent automatic colorization. The complete video results are presented in the supplementary material.4.5.1 Ablation study of the GCG module. In our model, the GCG module enables the mapping of grayscale frames to color reference frames in a one-to-one or one-to-many manner. In this ablation study, we assess the impact of the GCG module itself, as well as its internal components, Color Guidance Attention (CGA) and Color Distribution Gated Attention (CDGA), on the model's performance. We train the following variants: (1) removing the influence of the GCG module entirely by concatenating grayscale Table 1 : Quantitative comparisons of our proposed method against state-of-the-art approaches on the synthetic LDV 3.0 test set and the DAVIS-Test-Dev 2017 test set. Here," Ours " represents our final model, " Ours* ", " Ours** ", and " Ours*** " respectively denote the ablation results without CGA and CDGA, with CGA and without CDGA, and without SRG.Deoldify [2] TCVC [38] VCGAN [70] DRemaster [17] DExample [63] (3) utilizing the complete GCG module, i.e., w/ CGA & w/ CDGA. We present our experimental results in Figure 5 and Table 1 . Observations from (1) show that although the model roughly learns the distribution of color structures and performs well in terms of color structure, the target frames do not effectively learn the corresponding color distribution from the reference frames. Additionally, in (2), although the model can obtain color distribution from the reference frames in most structures, the lack of the CDGA module results in unsatisfactory coloring effects in the details of the images, as shown in the highlighted box in Figure 5 . In contrast, our model achieves adaptive coloring through the GCG module. This not only ensures stable color propagation but also compensates for deficiencies in color distribution within image details by utilizing a gating mechanism.To validate the effectiveness of SRG, we conducted quantitative and visual comparisons between the results obtained without SRG (i.e., using a single reference image) and those obtained with SRG. This comparison is depicted in Table 1 under the " Ours*** " column and illustrated in Figure 6 . When we use a single image as a reference, i.e., w/o SRG, the model may not obtain sufficient color information from the reference, leading to inaccuracies in automatically coloring areas not covered by the reference. For instance, in the upper example of Figure 6 , our chosen single reference image does not include the hand, while in the lower example, our chosen single reference image is primarily a frontal face. Therefore, they may exhibit anomalies in the coloring results when new areas appear because they cannot obtain complete color information from the reference. However, by using the Sliding Reference Grid, our model is able to extract as many color structural features as possible from the reference images and previously colored frames, resulting in more natural and continuous outcomes. More details on the ablation experiments are provided in the supplementary materials. 
conclusion: 
 In this paper, we propose a two-stage coloring pipeline based on the pre-trained T2I model, i.e., Keyframes Joint Coloring and Full Video Coloring, achieving stable and photorealistic video colorization. Our method introduces Gated Color Control module (GCG) to achieve high-quality coloring under various conditions and utilizes Sliding Reference Grid strategy (SRG) for more stable and precise results. Experimental results confirm its superiority over previous state-ofthe-art models in terms of color accuracy, richness, and temporal consistency for video colorization tasks. 
 
 Appendix:  


Decision: Reviewers acknowledged the novelty and good results of the paper. The rebuttal was successful, with three reviewers raising their scores, and the paper received all accept recommendations. The area chairs agree with this recommendation and are pleased to inform you that your paper has been accepted. The authors are advised to incorporate the reviewers' comments into the camera-ready version. Specifically, there is a remaining concern from Reviewer 4QBZ wrt need for an efficiency comparison and a test using the proposed method without additional prompt input.

Reviews: review: paper_summary: paper_summary: The authors propose a two-stage framework for video colorization, building upon a text-to-image generation model. The proposed GCG module is used for color propagation, and the SRG module is used to enhance local details. Quantitative and qualitative experiments demonstrate that the proposed framework achieves optimal or near-optimal performance.
strength_weakness: strength_weakness: The authors successfully applied a diffusion generative model to the video colorization task, achieving good quantitative results.
limitations: limitations: 1. In line 189, the authors mentioned evaluating temporal consistency, but it is unclear which specific metric was used for this evaluation.
2. Lack of efficiency analysis: The authors mention in the abstract that SRG is used to balance coloring quality and efficiency, but there seems to be no experimental data to support this claim.
3. In the quantitative experiments, the authors used prompts generated by BLIP to guide their method. It is unclear whether the baselines in the comparison can also understand textual prompts, which raises concerns about the fairness of the experiments.
paper_summary: paper_summary: The paper introduces a novel two-stage video colorization framework that leverages a pre-trained text-to-image (T2I) model. The method aims to produce high-quality, stable, and temporally coherent colorization for grayscale videos.

The authors propose a Gated Color Guidance module (GCG) that adaptively performs color propagation or generation based on the structural differences between reference and grayscale frames. The first stage of the framework involves autonomously coloring keyframes under a reference-mask condition, with an emphasis on temporal coherence through an enhanced attention mechanism. The second stage employs a Sliding Reference Grid strategy (SRG) to extract color features from multiple frames, leading to more stable coloring.

However, I still have some concerns; please see the Limitations section.
strength_weakness: strength_weakness: The article presents several notable strengths in its approach to video colorization:

1. A significant advantage is the GCG module, which adaptively performs color propagation or generation based on the structural differences between reference frames and grayscale frames. This feature enhances the accuracy and stability of the colorization.

2. The SRG strategy is another advantage as it allows for the extraction of color features from multiple frames, contributing to the stability and quality of the colorization, especially in complex scenes.

3. The model emphasizes temporal coherence, which is crucial for video colorization, ensuring that the colors remain consistent across frames.

4. The method is evaluated using common datasets and metrics, which provides a thorough understanding of its performance and capabilities.
limitations: limitations: However, I still have the following concerns:

1. Why use image diffusion models as priors instead of video diffusion models? Considering that 3D diffusion models can generate more consistent results for videos, the explanation in the paper is insufficient.

2. Lack of discussion on the limitations of the method. For example, the performance of long video generation and issues like error accumulation are not addressed. There is also a lack of results for longer videos.

3. Some Implementation details are missing, such as the guidance scale in classifier-free guidance.
paper_summary: paper_summary: The paper introduces a novel method for video colorization with the Gated Color-Guidance module using Image Diffusion Models. In the training phase, the Gated Color Guidance module serves as an adapter in fine-tuning the pre-trained diffusion model. The inference phase is proposed with two-stage coloring strategy: the first stage is to autonomously colorizes keyframes and the second stage applies color using a reference-guided approach to achieve high-quality colorization. The method leverages diffusion models adapted for video, and the experiments verified the model’s superiority in color accuracy, richness, and temporal consistency for video colorization tasks.
strength_weakness: strength_weakness: - **Novelty**: The approach introduces a new Gated Color Guidance module for video colorization, and decompose the colorization steps into Keyframes Joint Coloring and Full Video Coloring, which achieve accurate colorization and stable control.
- **Theoretical and Technical Correctness**: The paper provides a clear and methodical description of the techniques used, including a Color Guidance Attention module and a Color Distribution Gated Attention module.
- **Adequate Evaluation**: The method is extensively evaluated against existing state-of-the-art models using qualitative and quantitative measures with two datasets, showing superior performance in terms of color accuracy and richness.
- **Clarity**: The paper is well-structured, with clear explanations and supportive figures that illustrate the effectively.
- **Applications**: The technique has practical implications for enhancing grayscale video content, potentially useful in media restoration.
limitations: limitations: - It is not clear of comparison or discussion on time cost to other methods, which is mentioned as a shortcoming in related works.
paper_summary: paper_summary: This paper introduces a Gated Color Guidance module (GCG), which is used to perform color propagation or generation according to the structural differences between reference frames and grayscale frames.
strength_weakness: strength_weakness: The paper proposes the Gated Color Guidance module (GCG) to generate color proposes a Sliding Reference Grid strategy (SRG), to capture information from more frames. The authors also use common datasets and evaluation metrics to evaluate the trained model.
limitations: limitations: Although the authors propose the GCG model, there are still some shortcomings in the paper.
1. The whole effect of the proposed method is not good enough. For example, in Figure 2, the color of the collar in the second and third rows on the left are unreasonable, and the colors of the eyes, nose and mouth in the second row are distorted.
2. Several kinds of mask are mentioned in the paper, but there are no reasonable explanations and comparisons, and there are no visual results. Therefore, it's not easy for readers to understand the meanings of these masks.