COCO-LC: Colorfulness Controllable Language-based Colorization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 16:10:04

Abstract: Language-based image colorization aims to convert grayscale images to plausible and visually pleasing color images with language guidance, enjoying wide applications in historical photo restoration and film industry. Existing methods mainly leverage large language models and diffusion models to incorporate language guidance into the colorization process. However, it is still a great challenge to build accurate correspondence between the gray image and the semantic instructions, leading to mismatched, overflowing and under-saturated colors. In this paper, we introduce a novel coarse-to-fine framework, COlorfulness COntrollable Language-based Colorization (COCO-LC), that effectively reinforces the image-text correspondence with a coarsely colorized results. In addition, a multi-level condition that leverages both low-level and high-level cues of the gray image is introduced to realize accurate semantic-aware colorization without color overflows. Furthermore, we condition COCO-LC with a scale factor to determine the colorfulness of the output, flexibly meeting the different needs of users. We validate the superiority of COCO-LC over state-of-the-art image colorization methods in accurate, realistic and controllable colorization through extensive experiments.

Main paper: 
 introduction: 
 Color plays a pivotal role in shaping human perception of the world. It serves not only as one of the most expressive visual elements, but also directly influences people's emotions through various color styles. Image colorization aims to convert grayscale images into color images, which has a wide applications in diverse fields such as old photo restoration, color grading, and automatic animation colorization. However, image colorization is an ill-posed problem as it involves inferring a three-channel color image from a singlechannel illuminance image, which may have multiple reasonable solutions. Despite many automatic colorization methods [9, 19, 23, 25, 35, 47] have been proposed, most of them suffer such color ambiguity, leading to under-saturared colors with the mean of all possible color choices. To address this issue, conditional colorization begins to attract researchers' interests. By imposing additional constraints, such as language [5-7, 16, 41, 45] , scribble [16, 44] , reference image [27] and palette [38, 42] , the method can more accurately render specified colors.Recent advancement on large language models (LLMs) and diffusion models empowers image processing with language guidance. Compared with other visual conditions, text descriptions are highly informative, and are simple and efficient to use. Recent language-based image colorization methods [5, 7, 16, 41] either train language-vision aligned models supervised by dense human annotations, which may overfit on a small dataset and result in poor quality; or employ pretrained large models such as BERT [11] and CLIP [31] to align language domain and image domain, which, however, is less effective in building an accurate correspondence between gray instances and text description.Based on the above analysis, we summarize three key requirements lies on the image colorization problem: (1) Realism. The colorization results should look realistic. (2) Consistency. The colorization results should match the semantic content of the original gray images. (3) Controllability. Users can flexibly adjust the color of the output. Existing methods can only fulfill one or two requirements, hardly satisfying all above standards: The unconditional ones provide no controllability and suffer under-saturation, while the conditional ones are less flexible or fail to render semantically correct colors, leading to color overflow and inconsistency.To build a powerful image colorization framework that simulatenously achieves realism, consistency and controllability, we propose COCO-LC for COlorfulness COntrollable Language-based Colorization. To strengthen realism, our key idea is a coarse-to-fine framework that leverages Stable Diffusion [33] , the pre-trained cross-modality generative model, to utilize its high capability for textual-visual modeling and powerful generative prior on the natural image distribution. To maintain the semantic consistency between gray inputs and colorization results, we further incorporate robust multi-level conditions with both low-level and high-level cues of the gray inputs. Finally, in order to provide more controllability, besides color types, we develop a novel colorfulness-controllable colorization decoder to produce results with diverse color styles ranging from vintage, realistic, to fantastic.Specifically, we propose a coarse-to-fine training framework. In the coarse colorization stage, we inject features from the largescale cross-modality model CLIP [31] into the latent space of the gray image. On this basis, the resulting latent codes with rich color information are used to guide the diffusion model to generate finelevel colorization results, which achieves accurate correspondence between the textual color words and visual gray instances.Additionally, we propose multi-level conditions that effectively alleviate color overflow and mismatch issues. We design a novel dual-branch feature extractor that aligns the feature granularity of the low-level edge features with diffusion features for balanced condition injection. A semantic-aware feature regularization is further proposed to provide high-level features to improve semantical correspondence.In terms of controllability, brightness, hue and saturation are three key characteristics to determine color values. While brightness is determined by the gray image and the control of hue is well studied for conditional colorization, controlling saturation remains less explored. To this end, we design a colorfulness-controllable colorization decoder, with a scaling factor to allow users to flexibly choose different color styles from vintage to gorgeous styles.With the novel designs above, COCO-LC comprehensively realizes realism, consistency, and controllability. Extensive experimental results demonstrate our superiority in generating high-quality color images over both automatic and language-based state-of-theart baselines. In summary, our contributions are threefold:• We propose a novel coarse-to-fine COCO-LC framework for language-based colorization that achieves high realism, consistency, and controllability simultaneously. We leverage cross-modality model CLIP to build accurate correspondence between the color words and gray instances, significantly improving the visual-textual color consistency. • We develop multi-level conditions with both low-level and high-level guidance to find accurate color-semantic correspondence. A novel dual-branch feature extractor is proposed to align the feature granularity for balanced condition injection, which effectively alleviates color overflow. • We design a colorfulness-controllable decoder, which adaptivelty fuses the predicted color information and the original grayscale information, allowing users to choose fantastic, realistic or vintage colorized results to flexibly satisfy diverse user preferences. 
related works 2.1 automatic colorization: 
 Automatic colorization aims to generate plausible colorful images from grayscale input without user guidance. Cheng et al. [9] introduced the first deep-based colorization model. In view of the imbalance of the color space, CIC [47] defines a classification loss in the quantized 𝑎𝑏 color space rather than the traditional regression loss, leading to more saturated results. Researchers produce models with more plausible colorization results with Pixcolor [12] . With the development of generative models, many researchers began to make efforts on colorization problem with GAN [10] . Cao et al. [4] conditioned GAN with grayscale information in multiple layers to maintain spatial feature consistency. ChromaGAN [37] adds color error and class distribution losses to optimize training for colorization. BigColor [23] improves vividness through pre-trained generative priors while suffering from color cast. DeOldify [1] utilizes U-Net as the generator model with a self-attention scheme and reduces direct GAN training time with a scheduled adversarial loss. Furthermore, to utilize the long range dependency and increase model capacity, transformer-based architectures [36] are widely used in colorization tasks [19, 20, 25] . ColorFormer [19] designs a color memory module to learn and store the semantic-color mapping. HistoryNet [20] introduces fine-grained semantic understanding and classification prior to achieve accurate colorization and prevent color overflow. Priors including class labels [18] , instance bounding boxes [35] , and semantic segmentation maps [49] are introduced to further guide colorization models with semantic information. However, unconditional colorization is an ill-posed problem, resulting in color ambiguity and under-saturation. From the users' perspective, lack of controllability limits the practicability of this kind of approaches.With the help of the flexibility of text descriptions, language-based colorization methods enable simple but effective control over instance colors. Unicolor [16] developed a unified framework to support colorization in multiple modalities, including text descriptions. It proposed a spatial partitioning heuristic method to align color words and instances by leveraging CLIP [31] as zero-shot classifier, but struggle with finer colorization ability in spatial, and can only deal with several classic colors. L-CoDe [41] introduced an colorobject correspondence matrix and an attention transfer module to address the color-instance coupling problem, along with a softgated injection module for resolving the color-object mismatch issue. L-CoDer [5] introduced transformers [36] into language-based colorization to deal with inaccuracy while keeping the language decoupling property. However, both L-CoDe [41] and L-CoDer [5] need to create an additional color-instance decoupling module to align colors with gray instances. This module have to train on extra human annotations, resulting in more training cost. Thus, L-CoIns [7] achieved instance awareness with the grouping mechanism to adaptively aggregate similar image patches without additional annotations. Furthermore, L-CAD [6] reduces the dependence of colorization on the precision of language guidance by using a pretrained diffusion model with rich color priors and superior cross-modal capability. However, it lacks the ability to generate high-resolution colorized images due to the cost of fine-tuning.Recently, diffusion model shows its advantages in image generation [32, 33] , image editing [34] and image restoration [29, 43] .As diffusion models hold high-quality color and semantic priors with the benifit from large-scale pretraining, L-CAD [6] designs a luminance-guided compression module and merge latent codes of grayscale images into diffusion process to maintain accurate structure when fine-tuning the Stable Diffusion [33] . Diffusing Colors [45] adopts the cold diffusion mechanism [2] to learn a colorization process iteratively. Besides, CtrlColor [28] develops multiple encoders based on ControlNet [46] according to different kind of conditions, such as user scribbles, reference images, regions or text descriptions. As colorized results need to spatially align with input grayscale images, diffusion-based colorization methods have to balance the trade-off between realism and consistency. Overall, diffusion-based image colorization is still a cutting-edge and challenging topic. 
coco-lc 3.1 preliminary: stable diffusion: 
 Diffusion models learn image distributions based on a diffusion process and a denoising process, where Stable Diffusion [33] operates in the latent space with a VAE encoder E and a VAE decoder D.During the diffusion process, random Gaussian noises 𝜖 ∼ N (0, 𝐼 ) are gradually added to the encoded latent feature 𝑧 0 = E (𝑥 0 ) of the input image 𝑥 0 in 𝑇 steps,EQUATIONproducing a series of noisy samples 𝑧 1 , ..., 𝑧 𝑇 . As 𝑧 𝑇 can be treated as a standard Gaussian noise approximately when 𝑇 is large enough, the denoising process can recover a realistic image 𝑥 0 = D (𝑧 0 ) from a srandard Gaussian noise iteratively, which is achieved by training a neural network U-Net 𝜖 𝜃 with parameter 𝜃 to predict 𝜖 at each timestep 𝑡 based on 𝑧 𝑡 with the loss function:EQUATIONwhere in Stable Diffusion, text conditions 𝑐 𝑡𝑒𝑥𝑡 are taken as guidance to constrain the generation through cross-attention mechanism. Besides text, Stable Diffusion can be addtionally conditioned on images with ControlNet [46] . ControlNet is a trainable copy of the diffusion model, serving as a side branch to accept and apply image conditions 𝑐 𝑖 to the main diffusion branch. The overall learning objective can be formulated as It is natural to apply ControlNet to the colorization task, i.e., using grayscale image 𝐼 𝑔𝑟𝑎𝑦 as 𝑐 𝑖 to predict the corresponding colorful image 𝐼 . However, we experimentally found that vanilla Control-Net is not competent to offer high-quality gray image constraints. ControlNet uses simple convolution layers to preprocess the conditional images, leading to imbalanced feature granularity between the ControlNet and diffusion branches and causing structure distortion. It also fails to build robust correspondence between the gray instances and the color words, resulting in color mismatch and color overflow. This paper propose a new croase-to-fine framework with novel multi-level consistency-aware conditions to solve the above problems.EQUATIONCOCO-LC takes as input a grayscale image 𝐼 𝑔𝑟𝑎𝑦 , a text prompt describing the desired color of the instances in 𝑐 𝑡𝑒𝑥𝑡 , and a scaling factor 𝛼 to indicate the targeted colorfulness, and produce a corresponding colorization result Î . As shown in Fig. 2 , the proposed COCO-LC framework consist of three key components: (1) coarseto-fine colorization; (2) consistency-aware multi-level condition;(3) colorfulness controllable colorization decoder.Our coarse-to-fine colorization (Sec. 3.3) first colorizes the latent feature 𝑧 𝑔𝑟𝑎𝑦 = E (𝐼 𝑔𝑟𝑎𝑦 ) with our proposed lightweight Colorful Information Adaptor (CIA). CIA leverages the power of CLIP [31] to create a coarsely colorized feature based on 𝑐 𝑡𝑒𝑥𝑡 and 𝑧 𝑔𝑟𝑎𝑦 . The resulting ẑ is fed into the main diffusion branch by our condition injection branch for fine-level colorization.For consistency-aware multi-level condition, we extract the edge map and segmentation map from 𝐼 𝑔𝑟𝑎𝑦 to serve as low-level structure and high-level semantic guidance, respectively. We merge the low-level edge maps and high-level semantic segmentation maps with the condition injection branch, as will be detailed in Sec. 3.4.In Sec. 3.5, we will introduce our colorfulness controllable colorization decoder (COCO-decoder), that adapts the VAE decoder D to merge grayscale image and provides a scaling factor 𝛼 to enable users to flexibly adjust the colorfulness of the output.While Stable Diffusion demonstrates satisfying performance of matching color words and image instances with cross-attention mechanism, it is not trival to find proper correspondence when the instances lie in the gray feature space because of the domain gap between the grayscale images and the color images. Our coarse-tofine colorization framework solve this issue by gradually narrow the domain gap in two stages. We utilize six ResBlocks to merge grayscale structure information and color style information hierarchically, following a paramerized version of AdaIN [15] . be formulated by Eq. ( 4):EQUATIONwhere 𝐹 𝑠𝑐𝑎𝑙𝑒 , 𝐹 𝑠ℎ𝑖 𝑓 𝑡 denote the mapping networks in CIA to wrap the color information into scale-shift parameters. 𝑧 is the structure feature initialized with E (𝐼 𝑔𝑟𝑎𝑦 ). 𝜇 (𝑧) and 𝜎 (𝑧) denote the mean and standard deviation of 𝑧. This fusion operation can be also understood as a parameterized version of AdaIN [15] , and is applied to 𝑧 six times. The training objective of CIA isEQUATION)where VGG_Loss is the perceptual loss [21] and we simply set 𝜆 = 1.During testing, as the color image 𝐼 is unavailbale, we utilize CLIP text encoder 𝐸 𝐶𝐿𝐼 𝑃 𝑇 and replace 𝑓 𝐼 with the encoded text description 𝐸 𝐶𝐿𝐼 𝑃 𝑇 (𝑐 𝑡𝑒𝑥𝑡 ), thanks to the alignment of the CLIP visual and textual spaces. Moreover, excluding other information that is not related to color in image features (such as texture, luminance), 𝑐 𝑡𝑒𝑥𝑡 with color cues has greater color information density, further enhance the reasonability of the coarse colorization results. Thus, we have ẑ = CIA(E (𝐼 𝑔𝑟𝑎𝑦 ), 𝐸 𝐶𝐿𝐼 𝑃 𝑇 (𝑐 𝑡𝑒𝑥𝑡 )). We also try to use 𝐸 𝐶𝐿𝐼 𝑃 𝑇 (𝑐 𝑡𝑒𝑥𝑡 ) during training CIA, but find it become hard to converge, likely due to the high sparsity of the language domain.mations of image in a finer granularity, thus the coarse colorization results only provide rough color-instance correspondence. Then in the fine stage, on the basis of the semi-colorized latent ẑ, we use the diffusion model to match the color words and instances precisely. The fine-stage objective function isEQUATIONwhere 𝑧 𝑒𝑑𝑔𝑒 and 𝑓 𝑠 denotes edge and semantic map feature conditions extracted from 𝐼 𝑔𝑟𝑎𝑦 , which will be introduced in Sec. 3.4. dealing with gray image conditions, since grayscale images hold rich structure and semantic information than simple conditions like depth maps, and Canny edge maps [46] . Vanilla ControlNet do not have the ability to extract enough semantic features to precisely control the diffusion generation, leading to color overflow, color mismatch and structure distortion.To balance the feature granularity between condition injection and diffusion generation, we propose a dual-branch feature extractor based on E. Given a grayscale image 𝐼 𝑔𝑟𝑎𝑦 , we utilize SAM [24] as a zero-shot edge detector to extract its instance-aware edge map 𝐼 𝑒𝑑𝑔𝑒 , and encode it into the latent space 𝑧 𝑒𝑑𝑔𝑒 = E (𝐼 𝑒𝑑𝑔𝑒 ). As mentioned before, we have inserted color information into 𝑧 𝑔𝑟𝑎𝑦 with CIA, to obtain ẑ with rich color priors. We concatenate ẑ and 𝑧 𝑒𝑑𝑔𝑒 , and fuse them using a single convolution layer. We use z = 𝐶𝑜𝑛𝑣 ( [ ẑ, 𝑧 𝑒𝑑𝑔𝑒 ]) as the input of the ControlNet branch.To provide highlevel feature regularization to build more accurate correspondence between the color and the instances, we leverage Mask2Former [8] to predict semantic segmentation maps based on 𝐼 𝑔𝑟𝑎𝑦 . As the semantic map assigns labels to each pixel, we treat each label as a text description and use CLIP text encoder 𝐸 𝐶𝐿𝐼 𝑃 𝑇 to extract the feature of each description appear in semantic maps. Finally, we aggregate these text features into a standard spatial semantic feature 𝑓 𝑠 ∈ R 𝑐 ×ℎ×𝑤 , where 𝑐 represents the dimension of CLIP text features and ℎ × 𝑤 is the resolution of 𝐼 𝑔𝑟𝑎𝑦 .We design a semantic feature fusion block (SFF) to modulate the original features 𝑓 𝑜𝑟𝑖 on the skip connection of U-Net with 𝑓 𝑠 , as illustrated in Fig. 4 . SFF adapts SPADE [30] with minor changes, and uses two submodules 𝐹 1 and 𝐹 2 to process and fuse the features in a "scale-shift" manner. This fusion operation can be formulated asEQUATIONwhere 𝐿𝑁 is the Layer Normalization. 𝐹 1 and 𝐹 2 denote two mapping network to wrap 𝑓 𝑠 into scale-shift parameters, with some convolution layers and downsampling layers.To generate colorization results with varying color richness according with the user preference, we present colorfulness controllable colorization decoder D COCO (COCO-decoder), based on the VAE decoder D with a scaling factor 𝛼. As shown in Fig. 5 , D COCO maintains a fixed decoder D, a trainable decoder D and trainable zero-initialized convolution layers 𝐹 0 . We feed the gray image latent feature 𝑧 𝑔𝑟𝑎𝑦 into D and get a set of middle features {𝑑 𝑖 } as the structure guidance, where 𝑖 is the layer index. Correspondingly, we feed the diffusion output 𝑧 0 in the fine colorization stage into D, EQUATIONNote that we use zero-initialized convolution layers to warm up the middle features during injection, preserving the capability of the pre-trained VAE decoder.In the inference phase, we can use 𝛼 to control the injection of gray information, i.e., we can get diverse colorfulness results ranging from fantastic and gorgeous (low 𝛼), bright and realistic (middle 𝛼) to grayish and vintage (high 𝛼), enabling users to choose the best colorized result according to their preference. 
experiments 4.1 implementation details: 
 4.1.1 Training. We train CIA with a single NVIDIA RTX 2080Ti GPU for 280k iterations with a batch size of 24. After that, we train COCO-LC on a single NVIDIA RTX 4090 GPU for 150k iterations with a batch size of 16. We use AdamW optimizer with 𝛽 1 = 0.9 and 𝛽 2 = 0.999. The learning rate is set to 10 -5 .We use a single NVIDIA RTX 2080Ti GPU for inference. All testing images are resized to 512 × 512 with bilinear interpolation. Besides, we transform the output image to 𝐿𝐴𝐵 space and replace its 𝐿 channel with that of 𝐼 𝑔𝑟𝑎𝑦 , to maintain the structure consistency. Our method will generate colorized results of the input grayscale images based on users' text description. If users don't provide any text descriptions, we utilize BLIP [26] to get a standard text description and do colorization automatically.We conduct our experiments on languagebased colorization datasets proposed by L-CoDe [41] and L-CoIns [7] : (i) the extended COCO-Stuff dataset, which is built upon the COCO-Stuff dataset [3] by discarding unqualified samples for the colorization task. We further filter out some black and white photos, remains 56K training images and 1.9K evaluation images; (ii) multi-instances dataset which includes multiple instances with different visual characteristics within a single image, including 65K training images and 7K evaluation images. For both datasets, each image is accompanied by a corresponding language description. 4.2.2 Evaluate data. Apart from the validation set of the above two training datasets which totally contain 6.9k images, we use the first 5k images in ImageNet validation dataset, i.e. ImageNet-val5k. For the extended COCO-Stuff dataset and multi-instances dataset, each image is accompanied by several corresponding language descriptions. For ImageNet-val5k, we use BLIP [26] to generate a default text description of the color image to evaluate our method.Following UniColor [16] , we use Frechet Inception Distance (FID) [14] and colorfulness [13] to quantitatively evaluate the quality of our colorization results. We also utilize Δcolorfulness that computes the absolute difference of ground truth and colorization results to indicate realism in advance. Moreover, to evaluate the controllability of text descriptions, we calculate CLIP similarity score [31] between the prompts and colorization results. As image colorization is an ill-posed problem which may have multiple reasonable solutions, we do not adopt Peak Signal-to-Noise Ratio (PSNR) [17] , Structural Similarity Index Measure (SSIM) [39] or Learned Perceptual Image Patch Similarity (LPIPS) [48] for precisely match pairs of colorized result and ground truth.We make comparisons with both automatic colorization methods and language-based coolorization methods. For automatic colorization methods, we compare our method with CIC [47] , Col-orFormer [19] , DDColor [22] and CT2 [40] . For language-based colorization methods, we compare our methods with UniColor [16] , L-CoDe [41] , L-CoDer [5] , L-CoIns [7] and L-CAD [6] .As shown in Fig. 6 , we make comparisons with previous language-based colorization methods. Our method reduces color overflow better and generates more bright and colorful results. While L-CoDe [41] , L-CoDer [5] , L-CoIns [7] may suffer from color overflow of yellow sign on the forth row, or struggle with color under-saturation, UniColor [16] produces unreasonable colors (e.g.the green meat in the third row) when conditioned by complex text descriptions and inaccurate colors (e.g.brown ties in the second row should be red in the text description). L-CAD [6] can generate globally natural images, but it fails to reduce color overflow in some local area, such as the red tennis racket in the first row, the red color of tie overflows to the white shirt in the second row, and the unreasonable magenta knife in the third row. Additionally, although L-CAD utilizes diffusion model to generate high-realistic results, it still suffers from grayish and unnatural results, which can be seen obviously in the last row.The green leaf and trees behind two cats are recognized by our model and colorized to proper colors, while other methods can only generate gray or inproper colors.As presented in Table 1 , we present three different variants with different scaling factors to generate different styles of colors, ranging from fantastic (𝛼 = 1), realism (𝛼 = 0.9) to vintage (𝛼 = 0.8). We make comparison with four unconditional method, CIC [47] , ColorFormer [19] , DDColor [22] and CT2 [40] to demonstrate our superior visual effect. The comparison between other five language-based colorization methods: Unicolor [16] , L-CoDe [41] , L-CoDer [5] , L-CoIns [7] and L-CAD [6] .The results indicate our method can produce more plausible and accurate results. It is noticable that our method can not only generate high-realistic colorization results with the lowest FID and Δcolorfulness, but also generate the most colorful results with the highest colorfulness, satisfy the appetites of different users.We further conduct user studies to evaluate the subjective perception of human observers. We invite 46 volunteers to answer 10 questions, each questions contains a text description and eight colorized results of previous methods and our results. We encourage participants to evaluate those colorized results from the following three aspects: (1) consistency with text descriptions; (2) realism of images; (3) personal preference. We present our results of different colorfulness ranging from fantistic, realistic, to vintage at the same time to evaluate the robustness of our COCO-Decoder. The statistics results are presented as Fig. 8 , which shows our method is prefered by most users.We conduct other four baselines to demonstrate effectiveness of our coarse-to-fine framework and multi-level condition injection.The colorization results can be seen in Fig. 9 . vanilla ControlNet. We train a vanilla ControlNet without any design. We find experimentally that vanilla ControlNet will lead to color incomplete and color overflow. As shown in Fig. 9 , tiny yellow spots appear on the bus body, and the number on the top of the bus is not completely colorized. The cloth of bus driver becomes purple due to the color overflow.Colorful Information Adaptor. We remove CIA which insert CLIP color priors to latent codes of gray images. Without CIA, it's more difficult to build the accurate correspondence between color words and gray instances, so that the results become grayish or out of the control of text descriptions. In Fig. 9 , the purple of the bus overflows to the light and the front glass, and the banana fails to create a link with "green" constaint.Semantic Feature Fusion. We disable the insertion of the spatial semantic feature extract by Mask2Former [8] and CLIP [31] . As a result, obvious color overflow occurs in the colorized images. In Fig. 9 , the bush behind the bus becomes purple and the paper sign under the bananas become green unexpectly.Low-level edge condition. We disable the low-level edge condition, leading to color overflow and color incompletion.COCO-Decoder. We present some results of different color styles by our colorfulness controllable decoder with different scaling factor, as shown at Fig. 7 . Our method provides a user-friendly way of controllable color richness to generate diverse colors ranging from bland to gorgeous. Please refer to our supplementary materials for more visualization results.In summary, we propose a coarse-to-fine framework that use CIA to insert rich color priors conditioned by language prompts in the coarse stage. On the basis of this semi-colorized result, we use high-level spatial semantic features and low-level edge latent codes to constrain colors to the correct area spatially. Disabling any one of them will lead to obvious color overflow.Figure 10 : Failure case of our method when dealing with multiple color words in a text prompt. Prompt: "the blue cup, the yellow cup and the cyan cup". The middle cup is incorrectly rendered red. 
conclusion and discussion: 
 In this paper, we present COCO-LC, a novel coarse-to-fine framework that achieves COlorfulness COntrollable Language-based Colorization. We design a multi-level condition to reduce color overflow and COCO-Decoder to generate colorized results with diverse color styles flexibly. Extensive experiments demonstrate the superiority of COCO-LC over state-of-the-art image colorization methods in accurate, realistic and controllable colorization.Limitations and Future Work. Stable Diffusion utilizes CLIP to align text and image domain, which struggles with complex text descriptions. When there are multiple colors and instances, it is hard to find accurate correspondence, leading to color-instance mismatch, as shown in Fig. 10 . The color of the middle blue cup turns red unexpectly. In our future work, we would like to adopt more powerful cross-modality models and generative backbones to enhance the robustness of colorization.  
 
 Appendix:  


Decision: The reviewers acknowledged the good results. The initial scores are all mixed with Borderline Reject, Weak Reject, Borderline Accept, Weak Accept. The main concerns from all reviewers are regarding the effectiveness of the CIA module and the lack of ablation and evaluation metrics. The authors tried to respond to these concerns, but not all reviewers were satisfied with the rebuttal. All the scores remained the same after the rebuttal. After careful discussion and consideration, the ACs decided to accept this paper for its strength in the results in the colorization task.

The authors are required to consider all the concerns raised by the reviewers, include the information added in the rebuttal phase (e.g., PSNR / SSIM / LPIPS metrics, time cost, ablation of CIA, etc.), and add necessary studies to further analyze the modules. Regarding PSNR/SSIM, the method is less favored. The authors are suggested to provide analysis if they focus too much on colorfulness rather than faithfulness. Reviewer CSkw left one concern with the color bleeding issue. The AC has checked the results and confirmed it happens but not seriously enough. Color bleeding is a common issue in colorization; it might stem from segmentation or edge issues. The authors are encouraged to discuss and analyze how color bleeding is reliant on segmentation and edge as well.

Reviews: review: paper_summary: paper_summary: This paper introduces a COCO-LC model, which completes the text-controlled colorization process through a two-stage Coarse to Fine Colorization approach. Initially, it leverages the advantages of CLIP to train a Colorful Information Adaptor (CIA) for Coarse Colorization. Subsequently, it uses ControlNet for fine Colorization. Additionally, it employs Canny edge maps and semantic segmentation maps extracted from grayscale images to achieve Multi-level Condition constraints in colorization, thereby mitigating color overflow. Finally, a Decoder is designed to control color intensity.
strength_weakness: strength_weakness: 1. The proposed coarse to fine colorization framework addresses the image colorization problem by gradually narrowing the gap between the grayscale image domain and the color image domain through two stages.
2. It maximally utilizes the feature information contained in the grayscale images (such as Canny edge maps and semantic segmentation maps) for colorization. 
3. In the qualitative comparison results, the model demonstrates significant advantages in detailed colorization and text matching.
limitations: limitations: 1. This paper divides the colorization stage into Coarse Colorization and Fine Colorization phases. A major question arises: if there's a conflict in the colorization results between these two phases, which phase does the model's result align with? For instance, if the Coarse Colorization phase doesn't produce correct colorization results, does the subsequent Fine Colorization phase enhance the erroneous results, or does it primarily aim to correct them?

2. In line 165, it's mentioned that the model in the paper can ensure the process of "from vintage to gorgeous styles" by adjusting the Scaling factor 𝛼. Is this process merely adjusting the intensity of color features, as shown in Figure 7, or does it directly change the colorization style?

3. In line 438, it's stated that during the training of CIA, features  are obtained from the CLIP image encoder as f_I=E_I^CLIP (I). However, during testing, features E_T^CLIP (I) obtained from the CLIP image encoder are used instead of f_I. Could this direct replacement lead to uneven feature distribution, considering that the text here mainly focuses on color-related text rather than the entire image?

4. There's a lack of contrast images showing the same grayscale image guided by different text instructions for different colors (only one case is presented on the first page, which is insufficient).

5. Evaluation metrics are missing. Although in section 4.2.3, it's mentioned that metrics like PSNR, SSIM, and LPIPS are omitted due to multiple possible solutions in image colorization, comparing these metrics for image quality and factuality after colorization is necessary, similar to the comparison results in L-CAD[1].

6. In the ablation experiments, it's not clear whether the colorization effect follows the Coarse to Fine stages; therefore, only results using CIA should be presented to clarify this.

[1]	Weng S, Zhang P, Li Y, et al. L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors[J]. Advances in Neural Information Processing Systems, 2024, 36.
paper_summary: paper_summary: The paper tackles language-based image colorization, which converts grayscale images into visually appealing color images with language guidance. The authors introduce a new framework , which improves image-text correspondence using a coarse-to-fine approach. It employs a multi-level condition leveraging both low-level and high-level cues from the grayscale image for accurate colorization without overflow. It also introduces a scale factor to control the colorfulness of the output.
strength_weakness: strength_weakness: The results presented in the paper demonstrate good visual performance.
The quantitative metrics demonstrate that the proposed method surpasses previous approaches.
limitations: limitations: 1. The role of the CIA module is technically concerning. As illustrated in Figures 2(a) and 3, the colors predicted by the CIA module are inaccurate. The original image depicts a purple bus, whereas the CIA predicts a red bus. How can features with such inaccurate representation enhance the alignment between color terms and the coloring results?
2. The intervals for the scaling factors shown in the Colorfulness Controllable Decoder (COCO-Decoder) are too small to observe any significant differences. Moreover, this method is very similar to the analysis of color properties in the latent space mentioned in the paper "Diffusing Colors: Image Colorization with Text Guided Diffusion", lacking innovation. Additionally, Figure 2 demonstrates that the COCO-Decoder does not require training, yet in Section 3.5, the authors mention a trainable decoder. This makes the pipeline of this module difficult to understand.
3. The innovation of the Consistency-aware Multi-level Condition module is also limited. The inclusion of edge information and semantic segmentation information has been widely used in previous coloring methods, and this paper does not propose insightful improvements.
4. The fairness of the comparisons with other methods is concerning. In Figure 6, the results labeled "Ours" have noticeably higher resolution than the results of other methods. Moreover, the L-CAD mentions that all comparisons were conducted at a resolution of 256, while this paper states that the test images were scaled to a resolution of 512. Is this the reason why the proposed method outperforms the compared methods?
paper_summary: paper_summary: This paper focuses on Language-based image colorization. A method, named COCO-LC, consists of coarse-to-fine colorization, multi-level condition and colorfulness controllable colorization is proposed. Quantitative evaluation, user study and visual samples are provided
strength_weakness: strength_weakness: - The proposed method is well motivated and reasonable, the paper is well-written and easy to follow.
- Quantitative evaluation, user study and visual samples are provided.
- The proposed method achieved good performance.
limitations: limitations: - The colorization requires several steps and may result in significant time costs, which is not discussed in the text.
- Since color guidance is given only through text prompts and a pre-trained CLIP is used, the effectiveness of the method seems to depend heavily on the ability of the CLIP to map colors, which has not been specifically fine-tuned. This may lead to unstable final generation results. In addition, text itself is a very coarse representation of color control, which may result in the method not applicable to fine control tasks.
- A typo in line 398: "adptor" should be "adaptor"
paper_summary: paper_summary: This paper introduces COCO-LC, a Language-based Colorization framework, which strengthens the image-text correlation by providing coarsely colorized results. The framework utilizes both low-level and high-level cues from the grayscale image to achieve precise, semantic-aware colorization without color overflow.
strength_weakness: strength_weakness: 1. The colorization results closely align with the semantic colors as described in the language.
2. The results look more realistic than other methods in Figure 6.
3. The experiments is convincing to me.
limitations: limitations: 1. It is unclear that the CLIP features of the color from the language can constrain the generation during the training process.
2. The two horses in Figure 7 are depicted with different colors, reflecting the description in the language as “two brown horses grazing in a field next to a tree.”
3. It lacks comparison of computational complexity.