L-C4: Language-Based Video Colorization for Creative and Consistent Colors

## Abstract

Abstract Automatic video colorization is inherently an ill-posed problem because each monochrome frame has multiple optional color candidates.
Previous exemplar-based video colorization methods restrict the user’s imagination due to the elaborate retrieval process. Alternatively, conditional image colorization methods combined with post-processing algorithms still struggle to maintain temporal consistency.
To address these issues, we present Language-based video Colorization for Creative and Consistent Colors (L-C4) to guide the colorization process using user-provided language descriptions.
Our model is built upon a pre-trained cross-modality generative model, leveraging its comprehensive language understanding and robust color representation abilities.
We introduce the cross-modality pre-fusion module to generate instance-aware text embeddings, enabling the application of creative colors.
Additionally, we propose temporally deformable attention to prevent flickering or color shifts, and cross-clip fusion to maintain long-term color consistency.
Extensive experimental results demonstrate that L-C4 outperforms relevant methods, achieving semantically accurate colors, unrestricted creative correspondence, and temporally robust consistency.

## 1 Introduction

Video colorization aims to convert monochrome videos into plausible colorful versions and is widely used in film restoration, artistic creation, and advertising. The objective of video colorization is to assign semantically accurate and visually pleasing colors while ensuring temporal consistency to maintain a smooth visual experience without flickering or color shifts. To achieve this goal effectively, researchers have explored various approaches, *e.g*., automatic video colorization methods (Lei & Chen, ; Thasarathan et al., ; Liu et al., ) that infer colors from monochrome semantic cues, exemplar-based video colorization methods (Zhang et al., ; Iizuka & Simo-Serra, ; Yang et al., ) that transfer provided exemplar colors to monochrome ones, and conditional image colorization methods (Chang et al., ; Huang et al., ; Chang et al., ) combined with post-processing algorithms (Lai et al., ; Lei et al., ; ) to remove flickering artifacts during the colorization process.

Although these advancements demonstrate great potential for video colorization in diverse applications, they still have several limitations due to their inherent task settings:
(i) Ambiguous color assignment.
Automatic video colorization methods face the ill-posed nature of the task, struggling when an instance has multiple optional color candidates. This can lead to colorization results that may not accurately meet users’ expectations (Fig. 1 first row).
(ii) Limited creative imagination.
Exemplar-based video colorization requires users to provide reference images. Since the process of retrieving appropriate references can be elaborate, and the retrieved images are generally collected in the wild, these methods may restrict the ability to assign creative colors to instances based on users’ imagination (Fig. 1 second row).
(iii) Vulnerable color consistency.
Although conditional image colorization methods combined with post-processing algorithms enable colorizing videos with more user-friendly interactive conditions (*e.g*., language descriptions), existing models still struggle to maintain temporal consistency when handling significant variations in object positions and deformations across frames (Fig. 1 third row).

In this paper, we propose an innovative language-based video colorization framework to understand user-provided language descriptions without requiring post-processing, effectively addressing the aforementioned issues:
(i) Semantically accurate colors.
Our model is built upon the cross-modality generative model (Rombach et al., ), leveraging its comprehensive language understanding and robust color representation abilities to assign semantically accurate and visually pleasing colors that meet users’ expectations.
(ii) Unrestricted creative correspondence.
We present the cross-modality pre-fusion module to generate instance-aware text embeddings. This module can correctly assign specified colors to corresponding instances within video clips, enabling the application of creative colors.
(iii) Temporally robust consistency.
To ensure robust inter-frame consistency, we propose temporally deformable attention, which lifts color priors from images to videos and prevents flickering or color shifts by effectively capturing each instance and maintaining similar feature representation.
Additionally, we introduce the cross-clip fusion to extend the temporal interaction scope, maintaining long-term color consistency when colorizing long videos.

We name our approach L-C4, the Language-based video Colorization for Creative and Consistent Colors. Compared to previous works, L-C4 offers the following advantages:
(i) Due to the flexible nature of language descriptions, users can explicitly colorize monochrome videos according to their instructions (Fig. 1 first row).
(ii) Using instance-aware text embeddings liberates the color-object correspondence from real-world constraints, enabling users to assign colors to each instance with creative colors (Fig. 1 second row).
(iii) The language-based video framework constructs spatial-temporal interactions, demonstrating robustness in maintaining color consistency with instances that are moving or deforming (Fig. 1 third row).
We summarize our contributions as follows:

- •
We propose a language-based video colorization model that enables user-friendly interactions and produces temporally consistent and visually creative colorization results.
- •
We present the temporally deformable attention and the cross-modality pre-fusion module to ensure inter-frame color consistency and instance-aware colorization, respectively.
- •
We introduce the cross-clip fusion that achieves long video colorization while maintaining color consistency during the inference phase.

Figure: Figure 1: Advantages of our language-based video colorization framework compared to relevant colorization methods (Liu et al., ; Yang et al., ; Chang et al., ): First row: Automatic methods cannot specify the color of each garment, whereas the language-based method explicitly establishes this correspondence to meet users’ expectations. Second row: Exemplar-based method cannot colorize the camel purple due to the difficulty of finding appropriate references, whereas the language-based method allows users to apply creative colors freely. Third row: Image colorization method combined with post-processing algorithms struggles to maintain color consistency when the fish swims rapidly, whereas the language-based method demonstrates greater robustness.
Refer to caption: x1.png

## 2 Related work

### 2.1 Video colorization

The video colorization task requires models to render semantically accurate and visually pleasing colors for targeted monochrome videos while maintaining temporal consistency across frames. Automatic video colorization methods are trained to infer colors from the semantic cues presented in monochrome video frames, without human intervention. Although researchers explore various approaches to improve colorization performance, *e.g*., self-regularization (Lei & Chen, ), generative adversarial learning (Zhao et al., ), and optical flow estimation (Liu et al., ), the problem remains inherently ill-posed.
To guide the colorization process, exemplar-based approaches (Wan et al., ; Zhang et al., ) gain significant attention, which leverage reference images or user-selected frames to construct implicit correspondences with exemplar features. DeepRemaster (Iizuka & Simo-Serra, ) introduces a fully 3D convolutional framework to extract temporal-spatial video features. BiSTNet (Yang et al., ) addresses issues of object occlusion and color bleeding by incorporating bidirectional feature fusion and semantic segmentation priors. While these methods can produce plausible results, they heavily rely on the relevance and quality of the chosen exemplars, limiting their applicability in diverse scenarios.

### 2.2 Language-based image colorization

Language-based image colorization aims to colorize monochrome images according to user-provided language descriptions, providing a flexible and user-friendly interaction approach.
Manjunatha *et al*. (Manjunatha et al., ) is the first to introduce this task, designing the feature-wise affine transformations (Perez et al., ) to inject language descriptions into the colorization process.
Similarly, LBIE (Chen et al., ) develops a recurrent attentive model to spatially fuse image and text features.
To ensure the colorization results accurately meet users’ expectations, L-CoDe (Weng et al., ) and L-CoDer (Chang et al., ) utilize additional annotated correspondences between object and color words, addressing the problem of color-object coupling.
Towards instance awareness, L-CoIns (Chang et al., ) and L-CAD (Chang et al., ) introduce a grouping mechanism and a sampling strategy, respectively.
Recently, researchers pay more attention to integrating multiple conditions within a unified framework (Huang et al., ; Liang et al., ), which further lowers the barrier for color assignment.
When it comes to colorizing monochrome videos, combining image colorization with post-processing algorithm (Lai et al., ; Lei et al., ; ) might seem reasonable but often leads to performance degradation in practical applications. Therefore, to guide the video colorization with language descriptions effectively, it is necessary to design a model tailored to this task.

### 2.3 Video diffusion model

Since diffusion models demonstrate dominance in image generation (Rombach et al., ; Zhang et al., ), researchers explore approaches to lift pre-trained image generative priors to video.
To overcome the challenges of modeling temporal dependencies, Ho *et al*. (Ho et al., ) extend denoising networks with 3D convolutions and train the model from scratch based on large-scale video datasets. However, this approach significantly increases computational resource demands. To address this, ControlVideo (Zhang et al., ) proposes a training-free strategy with inter-frame global attention. Other methods (Esser et al., ; He et al., ; Ho et al., ; Xing et al., ) focus on introducing temporal modules (*e.g*., temporal convolution (Carreira & Zisserman, ) and temporal attention (Bertasius et al., )) into existing image diffusion models (Rombach et al., ; Saharia et al., ). To preserve the generative priors of the pre-trained model, they only fine-tune the added corresponding module.
Recently, researchers turn their attention to generating long videos, exploring techniques like temporal co-denoising (Wang et al., ) and noise rescheduling (Qiu et al., ). Despite extensive research in video generation, existing methods still face challenges in maintaining the long-term color consistency of instances.

## 3 Method

In this section, we begin by providing an overview of our framework (in Sec. 3.1). Subsequently, we describe details of the temporal deformable attention that ensures inter-frame color consistency (in Sec. 3.2) and cross-modality pre-fusion module that generates instance-aware text embedding (in Sec. 3.3). Following this, we introduce the cross-clip fusion (in Sec. 3.4), designed to maintain long-term color consistency when colorizing long videos. Finally, we elaborate on details of network training (in Sec. 3.5).

Figure: Figure 2: The pipeline of L-C4. (a) During the training phase, video frames are projected into the latent space with a VAE encoder, and noise is subsequently added. The monochrome features $y^{\mathrm{lum}}$ extracted by the luminance (lum) encoder are added to the noised latent codes to align the global structure with the monochrome frames. We equip the denoising U-Net with the Temporal Deformable Attention (TDA) block, ensuring robust inter-frame color consistency. We present the Cross-Modality Pre-Fusion (CMPF) module to generate instance-aware text embeddings, enabling the application of creative colors for specified instances. (b) During the inference phase, we introduce the Cross-Clip Fusion (CCF) to maintain long-term color consistency when colorizing long videos. When decoding the predicted latent code $\tilde{z}^{0}$, multi-scale monochrome features from the luminance encoder are added into the corresponding scales of the VAE decoder through skip connections to preserve local details.
Refer to caption: x2.png

### 3.1 Overview

We illustrate the framework of L-C4 in Fig. 2.
As a latent diffusion model, L-C4 performs the forward and backward processes in the latent space. Specifically, given an $N$ frames video clip $X=\{x_{i}\}^{N}_{i=1}$, L-C4 adopts a pre-trained VAE encoder $\mathcal{E}$ to project each frame $x_{i}$ into latent code $z_{i}^{0}=\mathcal{E}(x_{i})$, and a pre-trained VAE decoder $\mathcal{D}$ to reconstruct the video frames as $\tilde{x}_{i}=\mathcal{D}(z_{i}^{0})$. The pre-trained weights of the VAE are obtained from SD1.5 (Rombach et al., ).

To preserve the global structure and local details of the monochrome video $X^{\mathrm{lum}}=\{x_{i}^{\mathrm{lum}}\}^{N}_{i=1}$, we additionally introduce a luminance encoder $\mathcal{E}^{\mathrm{lum}}$ that shares the structure with $\mathcal{E}$ to extract monochrome features as $y^{\mathrm{lum}}_{i}=\mathcal{E}^{\mathrm{lum}}(x^{\mathrm{lum}}_{i})$. This brings two advantages:
(i) The extracted features could align the global structure between the noised latent code and the monochrome frames. Specifically, they are added to the latent code after a convolution layer.
(ii) The multi-scale monochrome features could preserve local details during the latent code decoding. Specifically, we add them into the corresponding scales in the VAE decoder using skip connections. In practice, the pre-trained weights of $\mathcal{E}^{\mathrm{lum}}$ are obtained from L-CAD (Chang et al., ).

To maintain the temporally consistent representations across frames, we equip the denoising U-Net with Temporal Deformable Attention (TDA) (in Sec. 3.2).
The TDA is inserted between the spatial blocks (*i.e*., residual blocks and spatial self-attention blocks) and cross-attention blocks to capture the inter-frame dependency of hidden features.
To achieve the instance-aware colorization, we introduce Cross-Modality Pre-Fusion (CMPF) to generate the instance-aware text embeddings by improving the semantic representation of noun concepts (in Sec. 3.3). These embeddings are injected into the denoising U-Net via cross-attention blocks.
To colorize long videos, we propose Cross-Clip Fusion (CCF) to maintain long-term color consistency while reducing the computational consumption (in Sec. 3.4). CCF is only performed during the inference phase.

### 3.2 Inter-frame color consistency

To ensure temporally consistent representations across frames and avoid flickering or color shifts, we propose the Temporal Deformable Attention (TDA), illustrated as the green block in the denoising U-Net in Fig. 2 (a). Previous temporal attention (Blattmann et al., ; Esser et al., ; He et al., ) extracts context at fixed locations, while the global inter-frame attention (Zhang et al., ) may introduce irrelevant regions. Considering the assumption that colors in a video do not change dramatically over short periods, we compress the time dimension and adopt a deformable receptive field to capture dynamic features of instances in video colorization. Our proposed TDA could effectively capture each instance across frames while keeping their feature representations similar, regardless of significant variations in positions and deformation. As shown in Fig. 3, the TDA is calculated as the following steps:

(i) Reference points sampling.
Denote the hidden features as $h\in\mathbb{R}^{N^{\mathrm{f}}\times H\times W\times C}$, where $N^{\mathrm{f}}$ represents the number of frames, $H$ and $W$ are the height and width, respectively, and $C$ means the number of channels. To effectively reduce the consumption of computation, we uniformly sample reference points in the video to construct the sampled coordinate sequence $p\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{N}^{\mathrm{f}}\times 3}$, where $N^{\mathrm{rs}}$ and $N^{\mathrm{rt}}$ are separately the sampling rates for resolution and time. As a result, the sampling resolutions are significantly reduced to $[\hat{H},\hat{W},\hat{N}^{\mathrm{f}}]=[H/N^{\mathrm{rs}},W/N^{\mathrm{rs}},N^
{\mathrm{f}}/N^{\mathrm{rt}}]$. These reference points represent a compressed spatiotemporal space of the video clip.

(ii) Offset estimation.
To calculate the relevant context for the frame sequence, we further estimate the offset for each reference point. Specifically, we project the frame sequence $h$ via the linear projection $W_{\mathrm{o}}$, following a lightweight 3D convolution network $\delta_{\text{offset}}(\cdot)$ to generate the offsets. To further ensure that the estimated context covers the overall semantics, we confine the range of offsets into $[-1,1]$ as $\Delta p=\tanh(\delta_{\text{offset}}(hW_{\mathrm{o}}))$.

(iii) Context extraction.
With the estimated offsets, we could extract relevant context with deformed points and transform them into keys $\tilde{k}_{i}$ and values $\tilde{v}_{i}$ via corresponding projection matrices:

$$ $\quad\tilde{h}=\phi(h;p+\alpha\Delta p),\quad\tilde{k}_{i}=\tilde{h}W_{\mathrm {k}}^{i},\quad\tilde{v}_{i}=\tilde{h}W_{\mathrm{v}}^{i},$ (1) $$

where $\alpha$ is a hyperparameter to control the range of candidate context, $i\in\{1,\dots,N^{\mathrm{h}}\}$ is the index of attention heads, and $\phi(\cdot\,;\cdot)$ is the function that weights estimated deformed points, formulated as:

$$ $\phi(h;(p_{x},p_{y},p_{t}))=\!\!\!\!\sum_{(r_{x},r_{y},r_{t})}\!\!\!\!g(p_{x}, r_{x})g(p_{y},r_{y})g(p_{t},r_{t})h[r_{t},r_{y},r_{x}],$ (2) $$

where $g(a,b)=\max(0,1-|a-b|)$ and $(r_{x},r_{y},r_{t})$ indexes locations of hidden features $h$.
Finally, TDA performs in a multi-head attention manner, formulated as:

$$ $\hat{h}=\mathrm{Concat}_{i\in\{1,\dots,H\}}\Big{(}\mathrm{Softmax}\big{(}(q_{i }\tilde{k}_{i}^{\top})/\sqrt{d}\big{)}\tilde{v}_{i}\Big{)}W_{\mathrm{h}},\quad q _{i}=hW_{\mathrm{q}}^{i},$ (3) $$

where $W_{\mathrm{q}}$ and $W_{\mathrm{h}}$ are learnable matrices and $d$ is the number of channels.

Figure: Figure 3: Illustration of TDA’s structure and the different receptive fields. Left: We uniformly sample reference points and estimate offsets for each point to calculate deformed points. After that, we could extract relevant context with multi-head attention. Right: Previous temporal attention only extracts context at fixed spatial locations across frames, struggling to find relevant context when objects move or deform (*e.g*., the plane’s tail wing). The global attention may introduce features from irrelevant regions (*e.g*., calculating all features) and bring excessive computational consumption. Our proposed TDA can accurately capture relevant context across frames via the estimated deformed points, effectively addressing the aforementioned limitations.
Refer to caption: x3.png

### 3.3 Instance-aware text embedding

Previous language-based image colorization model (Chang et al., ) achieves instance awareness by introducing extra instance segmentation annotations and modifying the sampling strategy of the denoising process. However, this faces challenges in tracking instances that may be occluded or move in and out of frames when colorizing monochrome videos. To avoid introducing external requirements while correctly assigning colors to corresponding instances in video clips, we propose the Cross-Modality Pre-Fusion (CMPF) module as shown in the pink section of Fig. 2 (a). This module generates instance-aware text embeddings by improving the semantic representation of noun concepts, ensuring creative colorization results.

Specifically, the CMPF module comprises $L$ fusion blocks, each consisting of a Multi-head Self-Attention (MSA), a Masked Cross-Attention (MCA), and a Feed-Forward Network (FFN). Denote CLIP embeddings (Radford et al., ) of user-provided language descriptions at the $l$-th block as $y^{\mathrm{tex}}_{l}\in\mathbb{R}^{N^{\mathrm{t}}\times C}$, where $l\in\{1,\dots,L\}$ and $N^{\mathrm{t}}$ is the length of language descriptions. We first adopt an MSA to learn the inter-word modification relationships between embeddings as:

$$ $\displaystyle\bar{y}^{\mathrm{tex}}_{l-1}=\mathrm{LN}\big{(}\mathrm{MSA}(y^{ \mathrm{tex}}_{l-1})+y^{\mathrm{tex}}_{l-1}\big{)},$ (4) $$

where $\mathrm{LN}$ is the layer normalization.
Next, using the monochrome features $y^{\mathrm{lum}}$ as the visual prompt, we extract compressed features based on the aforementioned TDA as $\hat{y}^{\mathrm{lum}}=\mathrm{TDA}(y^{\mathrm{lum}})$. We perform the MCA to establish the correspondences between embeddings $\bar{y}^{\mathrm{tex}}_{l-1}$ and the compressed monochrome features $\hat{y}^{\mathrm{lum}}$. Note that we apply a mask $M$ to exclude the color-related words (*e.g*., “red” and “white”), allowing the model to focus on the understanding of noun concepts:

$$ $\displaystyle\tilde{y}^{\mathrm{tex}}_{l-1}=\mathrm{LN}\big{(}\mathrm{MCA}( \bar{y}^{\mathrm{tex}}_{l-1},\hat{y}^{\mathrm{lum}},M)+\bar{y}^{\mathrm{tex}}_ {l-1}\big{)}.$ (5) $$

Following that, the FFN integrates the combined features as:

$$ $\displaystyle y^{\mathrm{tex}}_{l}=\mathrm{LN}\big{(}\mathrm{FFN}(\tilde{y}^{ \mathrm{tex}}_{l-1})+\tilde{y}^{\mathrm{tex}}_{l-1}\big{)}.$ (6) $$

After $L$ times iteration, CMPF generates the instance-aware text embeddings $y^{\mathrm{tex}}_{L}$. Before injecting these embeddings into the denoising network, we apply a learnable linear layer $W_{\mathrm{e}}$, initialized to zero, and add them with the initial CLIP embeddings $y^{\mathrm{tex}}_{0}$ to provide an efficient initialization:

$$ $\displaystyle y^{\mathrm{tex}}=W_{\mathrm{e}}(y^{\mathrm{tex}}_{L})+y^{\mathrm {tex}}_{0}.$ (7) $$

### 3.4 Long-term consistent inference

Existing methods struggle to colorize long videos consisting of hundreds of frames due to computational resource limitations. A trivial approach is to independently colorize multiple non-overlapping video clips and then stitch them together. Intuitively, this strategy fails to ensure color consistency across video clips. Some colorization methods (Lei & Chen, ; Yang et al., ) adopt autoregressive strategies by referring to previously colorized frames. However, this strategy is prone to error accumulation.
While some methods (Liu et al., ; Wan et al., ) improve temporal consistency based on optical flow, they heavily rely on the precision of flow estimation. Inspired by recent diffusion-based fusion mechanisms (Wang et al., ), we introduce the Cross-Flip Fusion (CCF) to effectively extend the temporal interaction scope and maintain long-term color consistency when colorizing long videos.

Instead of using a sliding window to capture local context, we use a skip window to capture long-term color dependency. Specifically, we select $N^{\mathrm{f}}$ frame video clips with different time intervals $d\in\{1,2,4,\dots,N^{\mathrm{d}}\}$, where $N^{\mathrm{d}}$ is the largest power of 2 that is less than the total number of video frames. To achieve a smooth temporal experience, we fuse these cross-clip features for each frame. This process is visualized in Fig. 2 (b), where frames within the same clip are represented by patches of the same color. Given that frames closer to the central frame of the clip tend to include more representative features, we further implement the distance-based weighted fusion to alleviate inconsistencies at clip boundaries, instead of direct averaging:

$$ $\hat{f}_{i}=\sum\limits_{j}\frac{\left({N^{\mathrm{f}}}/{2}-|i-c_{i,j}|\right) f_{i,j}}{\sum_{k}\left({N^{\mathrm{f}}}/{2}-|i-c_{i,k}|\right)},$ (8) $$

where $|\cdot|$ denotes absolute value function, $c_{i,j}$ represents the index of the center frame in the $j$-th video clip that contains the $i$-th video frame, and $f_{i,j}$ denotes the feature of the $i$-th frame extracted by the TDA block in the $j$-th video clip.

### 3.5 Learning and implementation details

We adopt the two-stage training strategy for our model. In the first stage, we train the denoising U-Net $\epsilon_{\theta}$ under the guidance of the CLIP embeddings (Radford et al., ) of language descriptions. In the second stage, we freeze the denoising U-Net and train the CMPF module to optimize the instance-aware embeddings. We apply the mean squared error (MSE) loss to optimize the model in both stages:

$$ $\mathcal{L}_{\mathrm{MSE}}=\mathbb{E}_{t,z_{0},\epsilon_{t}\sim\mathcal{N}(0,1 )}\big{[}\|\epsilon_{t}-\epsilon_{\theta}(z^{t},t,y^{\mathrm{tex}},y^{\mathrm{ lum}})\|^{2}\big{]}.$ (9) $$

Our model is trained on the subset of the InternVid dataset (Wang et al., ), which comprises 100K text-video pairs, and all samples have an aesthetic score above $5.5$.
We train the first stage over 20 epochs with a batch size of 2, spending approximately 100 hours. We use the AdamW optimizer with a learning rate of $1\times 10^{-5}$ and momentum parameters $\beta_{1}=0.99$ and $\beta_{2}=0.999$.
In the second stage, we use the same settings and train 10 epochs.
In this paper, we set the clip length for training $N^{\mathrm{f}}=8$ and hyperparameter $\lambda=1\times 10^{-4}$.
All experiments are conducted on 6 NVIDIA V100 graphics cards.

## 4 Experiment

**Table 1: Quantitative results on two evaluation benchmarks. Throughout this paper, $\uparrow$ ($\downarrow$) means higher (lower) is better. Best performances are highlighted in bold.**
| Method | DAVIS30 | Videvo20 |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Color. $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FVD $\downarrow$ | CDC $\downarrow$ | Color. $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FVD $\downarrow$ | CDC $\downarrow$ |  |
| AutoColor | 23.69 | 23.20 | 0.919 | 0.239 | 962.72 | 3.671 | 23.20 | 23.73 | 0.923 | 0.258 | 1381.01 | 1.960 |
| VCGAN | 14.26 | 24.26 | 0.927 | 0.235 | 1367.31 | 5.208 | 14.53 | 23.94 | 0.920 | 0.223 | 1351.53 | 3.753 |
| TCVC | 19.71 | 25.04 | 0.922 | 0.224 | 1143.53 | 3.855 | 19.74 | 24.84 | 0.929 | 0.219 | 975.41 | 1.956 |
| DeepExemplar | 23.84 | 24.51 | 0.905 | 0.230 | 1175.74 | 4.229 | 25.11 | 23.21 | 0.916 | 0.236 | 794.11 | 2.013 |
| DeepRemaster | 18.49 | 24.33 | 0.913 | 0.225 | 1073.95 | 5.400 | 20.93 | 23.40 | 0.904 | 0.208 | 883.05 | 3.437 |
| BiSTNet | 27.00 | 24.88 | 0.928 | 0.214 | 977.15 | 3.946 | 25.30 | 24.43 | 0.924 | 0.217 | 749.85 | 1.974 |
| L-CoIns | 17.06 | 23.04 | 0.871 | 0.236 | 1474.44 | 3.946 | 16.46 | 23.44 | 0.872 | 0.260 | 1387.66 | 2.027 |
| UniColor | 19.30 | 22.74 | 0.851 | 0.238 | 1245.69 | 4.569 | 18.71 | 23.04 | 0.857 | 0.246 | 1198.54 | 2.884 |
| L-CAD | 19.80 | 22.70 | 0.886 | 0.212 | 1347.26 | 4.407 | 21.98 | 23.30 | 0.881 | 0.221 | 962.72 | 2.945 |
| W/o TDA | 29.01 | 25.24 | 0.915 | 0.219 | 761.85 | 3.646 | 29.15 | 25.06 | 0.927 | 0.229 | 468.31 | 1.775 |
| W/o CMPF | 28.72 | 24.84 | 0.923 | 0.227 | 724.63 | 3.673 | 28.72 | 24.82 | 0.921 | 0.224 | 476.52 | 1.658 |
| W/o CCF | 29.19 | 25.23 | 0.927 | 0.216 | 735.76 | 3.593 | 29.51 | 24.74 | 0.935 | 0.204 | 496.81 | 2.335 |
| Ours (L-C4) | 29.33 | 25.69 | 0.933 | 0.209 | 654.32 | 3.114 | 32.59 | 25.17 | 0.939 | 0.198 | 420.59 | 1.572 |

Figure: Figure 4: Visual quality comparison with automatic video colorization methods (Lei & Chen, ; Zhao et al., ; Liu et al., ).
Refer to caption: x4.png

Figure: Figure 5: Visual quality comparison with exemplar-based video colorization methods (Zhang et al., ; Iizuka & Simo-Serra, ; Yang et al., ).
Refer to caption: x5.png

We perform comparisons and ablation studies on two widely used benchmarks: the DAVIS (Perazzi et al., ) and the Videvo (Lai et al., ) datasets. Adhering to the evaluation protocol of existing video colorization methods (Liu et al., ; Zhao et al., ; Yang et al., ), we conduct evaluation experiments on the DAVIS validation set (30 videos) and the Videvo test set (20 videos). Following the most relevant language-based colorization methods (Chang et al., ; Huang et al., ; Chang et al., ), we select a resolution of $256\times 256$ to ensure a fair comparison. During training, we leverage the language descriptions provided by InternVid Wang et al. (), which employs InstructBLIP (Dai et al., ) to generate descriptions for every 20-frame interval and then integrate these descriptions into a single coherent caption for each video using GPT-4 (Achiam et al., ). During evaluation, we recruit human volunteers to annotate language descriptions to evaluate the practical applicability.

### 4.1 Comparison with state-of-the-art methods

We make comparisons with 3 automatic video colorization methods (*i.e*., AutoColor (Lei & Chen, ), TCVC (Liu et al., ), and VCGAN (Zhao et al., )), 3 exemplar-based colorization methods (*i.e*., DeepExemplar (Zhang et al., ), DeepRemaster (Iizuka & Simo-Serra, ), and BiSTNet (Yang et al., )), and 3 language-based image colorization methods (*i.e*., L-CoIns (Chang et al., ), UniColor (Huang et al., ) and L-CAD (Chang et al., )) combined with the post-processing algorithm (Lei et al., ) to demonstrate the effectiveness of our L-C4 in the color consistency, instance awareness, and creative colorization. Note that only language-based colorization methods Chang et al. (); Huang et al. (); Chang et al. () share the same task setting as our approach. Other video colorization methods are presented to highlight the advantages of our task setting. All comparison experiments are conducted using the officially released code for each method.

Figure: Figure 6: Visual quality comparison with language-based image colorization methods (Huang et al., ; Chang et al., ) combined with post-processing algorithms (Lei et al., ).
Refer to caption: x6.png

Evaluation metrics.
At the frame level, we employ the colorfulness (Color.) score (Hasler & Suesstrunk, ) to assess the vividness of colors in the colorization results. Additionally, we report PSNR (Huynh-Thu & Ghanbari, ), SSIM (Wang et al., ), and LPIPS (Zhang et al., ) metrics to evaluate the perceptual difference between the colorized frames and the ground truth. At the video level, we utilize the Fréchet Video Score (FVD) (Unterthiner et al., ) to quantitatively assess the perceptual realism of the whole colorized videos, which calculates the feature distribution similarity between the colorization results and the ground truth.
We further measure the temporal consistency using the Color Distribution Consistency (CDC) index, reported with a scale of 1000 for better readability (Liu et al., ).

Quantitative comparisons.
As shown in Tab. 1, our method achieves the best scores across all evaluation metrics.
The best FVD score demonstrates that the overall quality of our colorization surpasses other methods.
With the CPFM that generates instance-aware text embeddings, our method ensures accurate colorization results for specified instances, resulting in higher PSNR, SSIM, and LPIPS scores.
Leveraging the robust color representation of the diffusion model, our method achieves more vivid colorization results with a higher colorfulness score.
Additionally, equipped with our proposed TDA and CCF, our colorization results present more robust temporal consistency, achieving the best CDC score.

Qualitative comparisons.
As shown in Fig. 4, L-C4 presents semantically accurate colors. On the top, with the prompt “blue/colorful”, the cups are vividly colorized. On the bottom, users explicitly assign the red/green color to the right person to meet their expectations.
As shown in Fig. 5, L-C4 achieves unrestricted creative correspondence. On the top, the oranges are colorized pink, a color hardly observed in nature. On the bottom, users can freely assign the cat’s appearance, eliminating the need for corresponding exemplars.
As shown in Fig. 6, L-C4 demonstrates temporally robust consistency. On the left, the color of the man’s hat remains consistently black across frames despite movement. On the right, the color of the dancing woman stays consistent.

### 4.2 User study

We conduct user studies to evaluate the subjective perception of human observers:
(i) Spatial Color Assignment (SCA) experiment: This experiment assesses whether our method assigns semantically accurate colors more effectively than relevant comparison methods. Participants are shown a random monochrome frame from the video along with 7 colorization results and asked to select the one that presents the best visual quality.
(ii) Temporal Color Consistency (TCC) experiment: This experiment determines whether our method provides more robust color consistency across frames compared to relevant methods. Participants are shown a monochrome video along with 7 colorization results, and instructed to select the one that offers the smoothest visual experience without flickering or color shifts.
For each experiment, we randomly select 10 samples from the testing videos of each dataset and let 25 volunteers make their choices independently on the Amazon Mechanical Turk (AMT). As shown in Tab. 2, L-C4 achieves the highest scores in both experiments.

Table: Table 2: User study results. The proposed method produces the highest scores in both experiments.

### 4.3 Ablation study

Figure: Figure 7: Ablation study results. When our proposed modules are disabled, the results exhibit vulnerable temporal color consistency and limited instance awareness.
Refer to caption: x7.png

We create three baselines to study the impact of our proposed modules. The evaluation scores and colorization results of each ablation study are presented in Tab. 1 and Fig. 7, respectively.

W/o Temporally Deformable Attention (TDA).
We replace the temporal deformable attention with the temporal attention that extracts context at fixed locations across frames. As a result, this variant struggles to maintain color consistency during instance movement and deformation (*e.g*., the color of the slate on the wall changes in the first row of Fig. 7, left).

W/o Cross-Modal Pre-Fusion (CMPF).
We use the text encoder of CLIP (Radford et al., ) to encode language descriptions, discarding our proposed cross-modal pre-fusion module. This leads to suboptimal performance in correctly assigning colors to corresponding instances based on users’ specific requests (*e.g*., the boy’s clothes turn green in the second row of Fig. 7, right).

W/o Cross-Flip Fusion (CCF).
We remove the cross-clip fusion and perform clip-by-clip inference when colorizing long videos. Consequently, this ablation fails to maintain color consistency across video clips, and instances not described tend to be colorized inconsistently (*e.g*., the sky changes from blue to purple and the chairs shift from gray to green in the third row of Fig. 7, left and right).

## 5 Conclusion

In this paper, we propose L-C4, an innovative framework for Language-based video Colorization for Creative and Consistent Colors. Compared to existing colorization methods that suffer from ambiguous instance correspondence, vulnerable color consistency, and limited creative imaginations, L-C4 understands user-provided language descriptions without requiring post-processing and effectively addresses the aforementioned issues.
To assign semantically accurate and visually pleasing colors that meet users’ expectations, we leverage the generative priors of a pre-trained cross-modality generative model.
To correctly assign specified colors to corresponding instances and enable the application of creative colors, we present the cross-modality pre-fusion module to generate instance-aware text embeddings.
To ensure robust inter-frame consistency and maintain long-term color consistency when colorizing long videos, we propose temporally deformable attention and cross-clip fusion.
Extensive experiments demonstrate the effectiveness of L-C4, achieving the best scores across six qualitative evaluation metrics on two widely used datasets.

## Appendix A Appendix

### A.1 Failure cases

Despite the various modules and strategies we have designed, our L-C4 still has difficulty distinguishing fine-grained colors. As shown in Fig. 8, our model cannot accurately colorize the car with “Klein blue” or “dark blue”. We will continue to explore precise color control in future work.

Figure: Figure 8: Failure cases in assigning fine-grained colors.
Refer to caption: x8.png

### A.2 Additional challenging dataset

We conduct an additional experiment to evaluate comparison methods on the randomly selected 30 videos from a subset of VRIPT (Yang et al., ), where all samples have aesthetic scores greater than 5. These videos do not overlap with the training data for any of the comparison methods. Quantitative comparisons are presented in Tab. 3. The results show that our method demonstrates superiority over the compared methods.

**Table 3: Additional quantitative results on the VRIPT dataset.**
| Method | VRIPT |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Color. $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FVD $\downarrow$ | CDC $\downarrow$ |  |
| AutoColor (Lei & Chen, ) | 20.77 | 22.48 | 0.903 | 0.274 | 2461.63 | 5.845 |
| VCGAN (Zhao et al., ) | 18.10 | 22.00 | 0.907 | 0.234 | 1885.02 | 5.256 |
| TCVC (Liu et al., ) | 22.86 | 23.48 | 0.918 | 0.204 | 1250.74 | 3.737 |
| DeepExemplar (Zhang et al., ) | 30.42 | 24.76 | 0.932 | 0.140 | 1040.08 | 4.050 |
| DeepRemaster (Iizuka & Simo-Serra, ) | 25.27 | 23.72 | 0.918 | 0.160 | 845.98 | 3.437 |
| BiSTNet (Yang et al., ) | 26.98 | 24.47 | 0.929 | 0.139 | 824.65 | 3.876 |
| L-CoIns (Chang et al., ) | 22.64 | 21.74 | 0.893 | 0.207 | 2389.62 | 4.649 |
| UniColor (Huang et al., ) | 24.26 | 21.17 | 0.875 | 0.205 | 2084.76 | 3.761 |
| L-CAD (Chang et al., ) | 25.29 | 23.94 | 0.904 | 0.183 | 1863.65 | 3.761 |
| Ours (L-C4) | 35.25 | 25.23 | 0.947 | 0.131 | 695.11 | 2.345 |

### A.3 Additional ablation results

We provide the following fine-grained quantitative ablation results to demonstrate the impact of our proposed modules, as presented in Tab. 4 (top):

3D convolution. Replacing the temporal deformable attention with standard 3D convolutions.

Removing mask. Removing the mask that excludes the color-related words in the MCA of CMPF.

Sliding window. Using a sliding window instead of a skip window to capture context in CCF.

Direct averaging. Use direct averaging instead of distance-based weighting to fuse clips in CCF.

Temporal L-CAD. Equipping L-CAD with temporal attention to create an intuitive baseline.

We further adjust the hyperparameters to show their impacts on L-C4, as shown in Tab. 4 (bottom):

Hyperparameter $\alpha$. Adjusting the range of candidate context used in context extraction of TDA.

Hyperparameter $N^{\mathrm{f}}$. Adjusting the number of clip frames used in the skip window of CCF.

**Table 4: Additional quantitative ablation results.**
| Method | DAVIS30 | Videvo20 |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Color. $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FVD $\downarrow$ | CDC $\downarrow$ | Color. $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FVD $\downarrow$ | CDC $\downarrow$ |  |
| 3D convolution | 28.45 | 25.42 | 0.924 | 0.211 | 774.61 | 3.893 | 28.99 | 25.09 | 0.930 | 0.234 | 472.53 | 2.130 |
| Removing Mask | 29.13 | 25.21 | 0.922 | 0.217 | 716.42 | 3.124 | 29.52 | 25.13 | 0.937 | 0.209 | 445.15 | 1.754 |
| Sliding window | 28.54 | 25.45 | 0.931 | 0.215 | 678.35 | 3.487 | 29.36 | 25.71 | 0.931 | 0.225 | 432.63 | 1.624 |
| Direct averaging | 29.09 | 25.13 | 0.927 | 0.209 | 695.42 | 3.593 | 29.51 | 25.58 | 0.938 | 0.205 | 464.38 | 1.939 |
| Temporal L-CAD | 29.32 | 24.99 | 0.914 | 0.239 | 779.34 | 3.859 | 29.52 | 25.06 | 0.918 | 0.239 | 552.12 | 2.047 |
| $\alpha$=2 | 29.12 | 25.26 | 0.929 | 0.224 | 694.34 | 3.135 | 31.35 | 24.92 | 0.915 | 0.223 | 487.75 | 1.624 |
| $\alpha$=8 | 28.32 | 25.62 | 0.917 | 0.216 | 662.63 | 3.536 | 31.56 | 25.22 | 0.924 | 0.236 | 449.41 | 1.834 |
| $N^{\mathrm{f}}$=4 | 28.45 | 25.50 | 0.921 | 0.225 | 667.36 | 3.354 | 30.45 | 25.42 | 0.925 | 0.215 | 460.98 | 1.858 |
| $N^{\mathrm{f}}$=12 | 28.24 | 25.62 | 0.921 | 0.217 | 673.73 | 3.145 | 31.45 | 25.43 | 0.932 | 0.225 | 482.67 | 2.053 |
| Ours (L-C4) | 29.33 | 25.69 | 0.933 | 0.209 | 654.32 | 3.114 | 32.59 | 25.65 | 0.939 | 0.198 | 420.59 | 1.572 |

### A.4 Additional visual quality comparison

In the main paper, we only show visual quality comparison with representative methods due to the space limit. As shown in Fig. 9, we show additional comparison results with automatic video colorization methods (*e.g*., AutoColor (Lei & Chen, ), VCGAN (Zhao et al., ), and TCVC (Liu et al., )), exemplar-based video colorization methods (*e.g*., DeepExemplar (Zhang et al., ), DeepRemaster (Iizuka & Simo-Serra, ), and BiSTNet (Yang et al., )), and language-based image colorization methods (*i.e*., L-CoIns (Chang et al., ), UniColor (Huang et al., ) and L-CAD (Chang et al., )) combined with the post-processing algorithm (Lei et al., ) to demonstrate our advantages.

In addition, although ColorDiffuser (Liu et al., ) also presents a language-based video colorization model, they do not release their code or provide a detailed technical description for reproduction. Therefore, we could only compare our results with the frames presented in their paper. Our method predicts colors with higher saturation (Fig. 10, top) and produces more accurate semantic colors (Fig. 10, bottom left) as well as better preservation of local structures (Fig. 10, bottom right).

As illustrated in Sec. 4.3, we create three baselines to evaluate the efficacy of our proposed TDA, CMPF, and CCF modules. We provide an additional visual quality comparison in Fig. 11.

### A.5 Additional application results

We employ our method to colorize old black-and-white films using language descriptions. As the colorization results presented in Fig. 12,
L-C4 demonstrates strong generalization capabilities, making the restoration process accessible to non-expert users. Additionally, we present colorization results of L-C4 applied to real-world videos in Fig. 13.

Figure: Figure 9: Additional visual comparison results with relevant video colorization methods.
Refer to caption: x9.png

Figure: Figure 10: Additional visual quality comparison results with ColorDiffuser (Liu et al., ).
Refer to caption: x10.png

Figure: Figure 11: Additional visual quality comparison results with created baselines.
Refer to caption: x11.png

Figure: Figure 12: Application in old black-and-white film restoration.
Refer to caption: x12.png

Figure: Figure 13: Application in real-world video restoration.
Refer to caption: x13.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to weakness.
soundness: 3
strengths: 1. The paper is well-organized and well-written.
2. The design of the components in the proposed method generally makes sense.
summary: This paper proposes a language-based video colorization model and employs temporally deformable attention and fusion module to handle inter-frame color consistency and instance-aware colorization.
weaknesses: Method Issues

1.	The proposed method can only process frames with fixed and low resolutions, specifically 256x256, making it incapable of handling high-resolution scenes. The results shown in the paper are also low-resolution, which might obscure some detailed issues. However, diffusion-based methods often struggle with details, so there are doubts about whether the proposed method can handle fine details effectively.

Performance Issues

1.	Lack of comparisons with state-of-the-art (SOTA) colorization methods. The comparisons shown in the paper omit more recent methods, such as ColorMNet[1] and DDColor[2].
2.	Some of the qualitative comparisons in the paper seem unfair and meaningless. For example, Figure 5 presents a comparison between the exemplar-based methods and the proposed text-based method under completely different conditions (e.g., the exemplar shows an orange fruit, while the text description specifies a pink fruit). Such comparisons lack significance, and the pink fruit in the figure also exhibits noticeable color overflow issues, where the pink in the highlight areas turns gray.
3.	Some of the videos in the demo still show color flickering and inconsistency issues, especially in small objects like shoes.
4.	Some of the displayed images still exhibit color bleeding issues.
5.	For diffusion-based methods, different seeds can lead to varying results. Does the proposed method perform consistently across different seeds? If not, what is the success rate of the proposed method?
6.	In video processing, processing speed is also essential. Please provide comparisons with the speeds of other video colorization methods.

[1] Yang, Yixin, et al. "ColorMNet: A Memory-based Deep Spatial-Temporal Feature Propagation Network for Video Colorization." ECCV, 2025.

[2] Kang, Xiaoyang, et al. "Ddcolor: Towards photo-realistic image colorization via dual decoders." ICCV. 2023.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to the Weaknesses section

**EDIT:**

While the proposed framework appears reasonable, the paper has several major issues:

- overstating its contributions, even in the rebuttal
- evaluation results of baseline methods largely differ from those in the original papers. For example, prior to revision, unconditional colorization/using ground-truth frames as exemplars, the reported metrics of baseline methods in this paper seem significantly worse. Additionally, the rough metric-based ranking of all baseline methods also differs from what is reported in other papers.

As a result, I have decided to lower my rating to <5
soundness: 2
strengths: # Strengths

- The design of proposed submodules are reasonable and technically sound
- Extensive experiments demonstrate the effectiveness of the proposed framework
summary: # Summary

The paper proposes a stable diffusion based video colorization framework. The paper leverages the temporal deformable attention for color consistency, cross-modality prefusion for instance-awareness, a skip window design for long-term temporal consistency. Experiments show the proposed method outperforms the selected baselines.
weaknesses: # Weaknesses

- The "Limited creative imagination" in L044 might not be a weakness of exemplar-based video colorization considering the user can use text-based image colorization methods to obtain exemplars
- It seems the paper overclaims its contribution in L091-092. It's unclear how reliable the proposed framework can "assign colors to each instance" considering the proposed framework still uses CLIP-based embeddings
- It seems that the design of LumEncoder is same as the previous works, but the paper does not cite them
- The paper does not mention how the exemplars were selected for the exemplar-based baselines. For L-C4 colorization with user prompts, the exemplar-based baselines might also need to adopt text-based colorization results as exemplar for fair comparisons
- Fig 4 and Fig 5 only contain one frame for each method, which seems inadequate to visualize the visual quality for video colorization. Visible color shifting artifacts can be observed in the supplementary video
- The paper does not mention the detailed evaluation settings: are prompts used in quantitative evaluations, if used, how they are obtained? 
- Diffusion models can generate diverse results. The paper does not mention how results are selected for comparison and does not evaluate the success rate for each generation
- CDC metrics seem to have different scales with previous works
- Considering there is no ground truth for video colorization, using PSNR/SSIM for evaluation seems strange
- The paper does not compare with recent text-based video colorization methods like [1] and [2]
- The scale of user study seems relatively limited
- In Fig 7 right part, the boy's shirt in L-C4 seems suffering from color bleeding issue
- The paper does not mention runtime statistics, how long does the proposed framework take to colorize a video with 300 frames?

- [1] Towards Photorealistic Video Colorization via Gated Color-Guided Image Diffusion Models
- [2] Versatile Vision Foundation Model for Image and Video Colorization

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness section.
soundness: 3
strengths: 1. Innovative Integration of Components: The combination of CMPF, TDA, and CCF addresses key challenges in video colorization, such as instance-aware color assignments and temporal consistency.
2. Comprehensive Evaluation: The paper conducts extensive experiments, including quantitative metrics, qualitative analyses, user studies, and ablation studies, providing a well-rounded assessment of L-C4's performance.
3. Clear Motivation: The authors effectively identify the limitations of existing video colorization methods and articulate how L-C4 overcomes these issues through its novel framework.
4. State-of-the-Art Performance: L-C4 achieves superior results on multiple benchmarks, indicating its effectiveness in real-world applications like film restoration and artistic creation.
summary: The paper introduces L-C4, a language-based video colorization framework designed to assign semantically accurate and creatively flexible colors to monochrome videos based on user-provided language descriptions. The proposed approach leverages a pre-trained cross-modality generative model and integrates three key components:

- **Cross-Modality Pre-Fusion (CMPF)**: Generates instance-aware text embeddings to facilitate precise color assignments based on language inputs.
- **Temporal Deformable Attention (TDA)**: Ensures inter-frame color consistency by dynamically capturing object features across frames.
- **Cross-Clip Fusion (CCF)**: Maintains long-term color consistency in extended video sequences by aggregating information across multiple clips.

Experimental evaluations on DAVIS, Videvo, and an additional VRIPT dataset demonstrate that L-C4 outperforms existing automatic, exemplar-based, and language-based image colorization methods across various quantitative and qualitative metrics. User studies further support the method's effectiveness in achieving temporally consistent and creatively satisfying colorizations.


**Post-rebuttal comment**: Having carefully considered the authors' rebuttal and fellow reviewers' perspectives, I maintain my recommendation of marginal accept. The authors have successfully clarified their contributions, which I appreciate. While there are some methodological limitations that remain - as noted by multiple reviewers including myself and acknowledged by the authors - I believe the work still makes a valuable contribution to the field. **These limitations, though noteworthy, do not overshadow the paper's positive aspects. Therefore, I support its borderline acceptance for publication.**
weaknesses: 1. **Incremental Novelty**: While L-C4 combines existing techniques in a new way, components like deformable attention and cross-modality fusion are based on established methods. The paper would benefit from highlighting more unique innovations or providing deeper theoretical explanations for the observed performance improvements.

2. **Model Complexity and Efficiency**: Integrating TDA and CCF adds significant complexity to the pipeline, resulting in longer inference times and larger model sizes compared to baseline exemplar-based methods.

3. **Performance vs. Efficiency Trade-offs**: The paper does not discuss the balance between L-C4's performance gains and its increased computational costs, which is important for evaluating its practicality in real-world applications.

4. **Artifacts from Cross-Clip Fusion**: The cross-clip fusion approach may introduce artifacts or inconsistencies at clip boundaries, especially in videos with rapid motion or sudden scene changes.

5. **Fairness in Comparisons**: Despite the authors' efforts to ensure fair evaluations, comparing different methods remains challenging due to varying inputs and guiding signals.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - What's the generation process of color-related masks? What if the text prompts are complicated or intricate? One example:
  "The sunset painted the sky with hues of burnt umber, casting a surreal glow over the landscape." Could the model handle these cases successfully, and to what extent will the quality and type of text prompts influence its performance?

- Temporally Deformable Attention (TDA) generates offset estimations for reference points. Would the performance get worse or better if we replace the offset estimation with an off-the-shelf model, such as an optical flow estimation model?

Thank the authors for the detailed explanation. My concerns have been fully addressed. Regarding the failure cases, I think that fine-grained color manipulation may be better controlled by other kinds of user interactions, such as reference images or strokes, given the high-level and semantic nature of text prompts. Overall, I think this is a paper that tackles text-guided video colorization problems with meaningful contributions and good demonstrations. I will raise my score.
soundness: 3
strengths: - Each component is well-motivated, with clear explanations of how they address specific challenges in video colorization. The paper is well-written, with logical organization and flow that makes it easy to understand.

- The proposed Cross-Modality Pre-Fusion (CMPF) module enhances text embeddings with instance-aware capability, ensuring that the objects are assigned with desired colors, enabling creative and precise colorization given user input.

- The Temporally Deformable Attention (TDA) and Cross-Clip Fusion (CCF) effectively maintain color consistency across frames in a video. These components help prevent flickering and color shifts, leading to smoother and more visually appealing results.

- The experiments are thourough and well-designed, demonstrating superiority over state-of-the-art approaches.
summary: The paper introduces L-C4, a novel framework for language-based video colorization that guides the colorization process of monochrome videos using natural language descriptions, addressing the primary challenges faced in video colorization tasks: ambiguous color
assignment, limited creative imagination, and vulnerable color consistency. The proposed framework builds upon pre-trained cross-modal generative models to utilize their strong language understanding capabilities. To ensure accurate instance-level colorization, the paper proposes the Cross-Modality Pre-Fusion (CMPF) module to enhance text embeddings with instance-aware capability. Deformable Attention (TDA) block and Cross-Clip Fusion (CCF) are designed to ensure temporal color consistency across long sequences. The paper is well-written with clear motivation and rationales. The experiments are thorough, demonstrating their effectiveness.
weaknesses: - Regarding the generation of instance-aware embeddings, the paper does not fully explain or justify the creation of color-related masks. This might limit the model's applicability in complex cases, such as when dealing with intricate or lengthy textual descriptions.

- The paper lacks discussions on limitations. 

- Some typos: Line 311, "cross-flip" should be "cross-clip".

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See weaknesses.

**Post rebuttal review:**

Thank you for your detailed response, which addressed some of my concerns. However, I still have several questions:

**W1:**

I am unclear on the necessity of employing **language-based video colorization**, given that diffusion models tend to be computationally intensive and slow, often requiring substantial GPU memory. Why not utilize **language-based image colorization** methods to generate high-quality exemplars and subsequently apply a lightweight exemplar-based video colorization approach? Such methods would be faster (32 seconds vs. 669 seconds) and yield superior results, as indicated by Tables 5 and 6.

**W2:**

The application of deformable convolution in video tasks has been extensively explored in prior research (e.g., Deformable ConvNets, and deformable attention in transformers). The proposed TDA module appears overly simplistic and lacks significant technical insights.

**W3:**

Regarding the Cross-Clip Fusion (CCF) strategy: Is it intended solely for sampling video clips? The authors state, "We introduce the Cross-Clip Fusion (CCF) to effectively extend the temporal interaction scope and maintain long-term color consistency when colorizing long videos." Have you tested this approach on long video sequences, such as those with a length of 500 frames? It seems that L-C4 may struggle to balance long-term performance with running time, as increasing ($N^f$) leads to a greater number of replicated features to fuse in CCF, resulting in longer inference time.

**W4:**

While the TDA and CCF modules focus on temporal coherence and CMPF is aimed at text-instance alignment, there appears to be a lack of technical insights related to color modeling from a colorization perspective.

**W5:**

This paper seems to extend a single image-based method to address the video colorization task. However, it does not adequately contextualize how existing methods manage temporal consistency and multi-modal alignment issues, nor does it clarify the true novelties of this work for evaluating its contributions.

**Quality-1:**

Table 1 indicates that on the DAVIS30 dataset, the configurations without TDA, CMPF, and CCF yield comparable CDC values. However, the integration of all three modules significantly improves the CDC by 0.479. Does this suggest that the video consistency is primarily derived from the network architecture rather than the proposed TDA? Additionally, Table 1 shows that on the Videvo20 dataset, the removal of CCF results in a degradation of CDC by approximately 0.8, indicating that CCF plays a critical role in maintaining consistency. Could you provide an explanation for this observation?

**Clarity:**

I appreciate that Figure 13 has already been updated for clarity.

**Final review:**

Thanks for the detailed response to my concerns

**W1:** I still have concerns regarding the evaluation results. The evaluations presented for both reference-based and non-reference-based metrics in Table 1 and Table 5 are confusing.
1. The Colorfulness score (Color. in Table 1) indicates the diversity of colors in the generated outputs, while PSNR, SSIM, and LPIPS measure fidelity to the ground truth. Typically, achieving high values in both metrics is challenging; for instance, a higher PSNR/SSIM/LPIPS often constrains the Colorfulness score due to the limited color range of the ground truth frames used for calculation. This phenomenon has been observed in numerous previous studies (e.g., Table 1 in the DeepExemplar paper, BiSTNet paper, and ColorMNet paper), where methods focusing on generating more colorful results, such as automatic image method, DDColor, achieve higher Colorfulness scores but relatively lower PSNR/SSIM/LPIPS. Conversely, exemplar-based methods prioritize fidelity, resulting in significantly higher PSNR/SSIM/LPIPS but lower Colorfulness scores (as demonstrated in Table 5 of this manuscript, comparing ColorMNet and BiSTNet). Have the authors assessed the Colorfulness score of the ground truth frames? It is unclear how a method can achieve both high PSNR and Colorfulness scores simultaneously.
2. Language-based methods assign colors based on descriptors (e.g., blue, white, green) but cannot specify exact hexadecimal values (e.g., #FFC0CB for pink). Unlike exemplar-based methods that utilize the first frame of the ground truth as an exemplar, language-based methods inevitably struggle with accurate color assignment and often cannot distinguish between similar colors (e.g., SeaGreen #2E8B57 and SpringGreen #3CB371). A slight error in color differentiation (such as mistaking SeaGreen for SpringGreen) can significantly impact PSNR, SSIM, or LPIPS outcomes. My primary concern is how L-C4, a language-based method, outperforms all exemplar-based methods (which use the first frame ground truth as an exemplar) in terms of PSNR, SSIM, and LPIPS.
3. The results for ColorMNet in Figure 16 are perplexing. The scenes depicted are relatively simple, characterized by straightforward objects and clean backgrounds. It is difficult to believe that ColorMNet fails to propagate color effectively, as shown with the camel in Figure 16, especially considering it utilizes DINOv2 to enhance feature extraction capabilities for more complex scenes. Furthermore, even the first frame—chosen as the exemplar—generated by ColorMNet exhibits significant color bleeding.

**W3:** 
My concerns remain inadequately addressed. Specifically, does L-C4 require more processing time when $N^f$ increases? It seems that L-C4 may struggle to balance long-term performance with inference time, as increasing $N^f$ results in a greater number of replicated features to fuse in CCF, leading to extended inference duration.

**Quality-1:**
The ablation study in Table 1 does not convincingly validate the effectiveness of the proposed modules. The authors' responses have not adequately addressed my concerns.

Given the aforementioned issues and the lack of novelty highlighted by other reviewers, I will change my rating to 3.
soundness: 3
strengths: 1. The proposed method is technically sound. 
2. Three main ablation results in Table 1 show the performance gains brought by the proposed techniques. 
3. The paper is generally in good shape.
4. The results in Table 1 and the video are impressive.
summary: The paper proposes a language-based video colorization method (L-C4) to colorize monochrome videos based on the user-input instructions. The proposed method contains four main components, a cross-modality generative model [28] for language understanding and color modeling, a temporally deformable attention for inter-frame consistency, a cross-modality pre-fusion module for instance-aware text embedding generation, and a cross-clip fusion for long-term color consistency. The proposed method was evaluated on two existing datasets and show better results against several existing colorization methods.
weaknesses: 1. Originality: The paper handles an existing task, video colorization, with an existing solution of using user-input languages as guidance [5,6,8,23,36]. The main difference is that while previous language-based methods are proposed for image colorization, the proposed method focuses on videos and considers modeling temporal consistency. However, the introduction does not discuss such differences and highlights the challenges and novelties. Comparing the language-based video colorization method to exemplar-based methods does not make sense, given the existence of language-based image colorization methods.

2. While the paper proposes a language-based video colorization method, the capability of understanding language and colors is from the existing image-based generation model [28]. Despite the proposed three methods (i.e., TDA, CMPF, and CCF), TDA and CCF focus on temporal coherency and CMPF is for text-instance alignment. There are no technical insights regarding color modeling from the colorization viewpoint.

3. The novelty of CMPF and CCF are not explained explicitly.

4. In fact, this paper is to extend a single image-based method [28] to handle the video colorization task. However, the paper does not provide a correct context (how existing methods handle temporal consistency and multi-modal alignment issues and what the true novelties of this work are) for evaluating its idea, and accordingly, the evaluations are not convincing (despite two experiments of L-CAD[4]+DF[20] and L-CAD[4]+temporal attention).

5. The authors claims that "Since the process of retrieving appropriate references can be elaborate, and the retrieved images are generally collected in the wild, exemplar-based video colorization methods may restrict the ability to assign creative colors to instances based on users’ imagination (Fig. 1 second row)". However, it is important to note that exemplar-based methods have the potential to incorporate language-based image colorization techniques, which can enable the assignment of creative colors. This raises the question of whether language-based video colorization methods are indeed necessary, or if the integration of language guidance into exemplar-based approaches could suffice to achieve the desired creative control over colorization. It would be beneficial for the authors to explore and discuss the limitations and advantages of both exemplar-based and language-based methods in the context of creative color assignment. A comparative analysis or experimental validation could strengthen their argument for the necessity of language-based video colorization methods over the adapted exemplar-based approaches.

Quality: (-) Most of the comparing methods are automatic and exemplar-based colorization methods, which are not close to the proposed method compared with those language-based image colorization methods [5,6,8,36,23] (the proposed method relies on the [28], which is also an image-based method). Meanwhile, there are also video diffusion models (as discussed in section 2.3) that study temporal coherency. Comparing the proposed method to (a combination of) language-based image colorization and video generation methods would be more convincing. (-) The proposed method lacks in-depth analysis and ablations, such as the sampling rate and hyperparameter alpha in TDA, the luminance encoder [6], and the hyperparameter N^f in CCF.

Clarity: (-) There are only some careless writings/typos: e.g., Temporal (Line 118) or Temporally (Line 54) Deformable Attention? Fig.12, last case (right): the colors should be red and blue. (-) The information about the training and evaluation datasets for existing methods and the proposed method is not clear, which may undermine the fairness of the comparisons.

Meta-review: The submission introduces L-C4, a language-based video colorization method. The reviewers appreciated the novel integration of components addressing temporal consistency and creative flexibility. However, concerns were raised regarding computational efficiency, limited comparisons with recent methods, and overstated contributions. While some reviewers acknowledged the rebuttal and saw merit in the work, others remained unconvinced due to unresolved issues like performance consistency and evaluation clarity. The paper shows promise but requires refinements. The scores received post-rebuttal are 3, 3, 6, 8 and 5. Given the mixed reviews, the AC recommends a borderline reject.

Decision: Reject