Title: L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors
Venue: NeurIPS 2023
Accepted: yes
Keywords: Colorization

Language-based generation

Diffusion model
Abstract: Language-based colorization produces plausible and visually pleasing colors under the guidance of user-friendly natural language descriptions. Previous methods implicitly assume that users provide comprehensive color descriptions for most of the objects in the image, which leads to suboptimal performance. In this paper, we propose a unified model to perform language-based colorization with any-level descriptions. We leverage the pretrained cross-modality generative model for its robust language understanding and rich color priors to handle the inherent ambiguity of any-level descriptions. We further design modules to align with input conditions to preserve local spatial structures and prevent the ghosting effect. With the proposed novel sampling strategy, our model achieves instance-aware colorization in diverse and complex scenarios. Extensive experimental results demonstrate our advantages of effectively handling any-level descriptions and outperforming both language-based and automatic colorization methods. The code and pretrained models
are available at: https://github.com/changzheng123/L-CAD.

L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors

Abstract

Language-based colorization produces plausible and visually pleasing colors under the guidance of user-friendly natural language descriptions. Previous methods implicitly assume that users provide comprehensive color descriptions for most of the objects in the image, which leads to suboptimal performance. In this paper, we propose a unified model to perform language-based colorization with anylevel descriptions. We leverage the pretrained cross-modality generative model for its robust language understanding and rich color priors to handle the inherent ambiguity of any-level descriptions. We further design modules to align with input conditions to preserve local spatial structures and prevent the ghosting effect. With the proposed novel sampling strategy, our model achieves instance-aware colorization in diverse and complex scenarios. Extensive experimental results demonstrate our advantages of effectively handling any-level descriptions and outperforming both language-based and automatic colorization methods. The code and pretrained models are available at: https://github.com/changzheng123/L-CAD .

1 Introduction

Image colorization is a challenging task that involves converting grayscale images into plausible and visually pleasing colorful ones. Language-based colorization approaches ( e.g ., [5, 6, 27]) use natural language descriptions as guidance to produce more controllable colorized images. These methods cater to users' specific requests, enabling them to provide more specific and nuanced color preferences. As automatic colorization ( e.g ., [39, 44, 52]) often encounters color ambiguity for common objects ( e.g ., flower colors), language-based colorization has shown promising results in producing high-quality and customizable colorized images with user-friendly input.

While language-based colorization methods have improved the consistency between language descriptions and colorization results through feature fusion [7, 27], decoupling color-object space [5, 45], and aggregating similar patches [6], they implicitly assume that users provide comprehensive color descriptions for most of the objects in the image. This assumption often leads to suboptimal performance, especially for objects without corresponding color descriptions. Furthermore, we have observed that users commonly assign colors only to their objects of interest.

L-CoDe

L-CoIns

L-CAD

A drink in a dark

purple mason jar

next to a bottle of

pink flowers.

Grayscale

Partial level

CT 2

DISCO

L-CoDe

L-CoIns

L-CAD

A colorful image.

Grayscale

Scarce level

CT 2

DISCO

L-CoDe

L-CoIns

L-CAD

The colors of the

wine are red, blue,

yellow, and cyan

from left to right.

Grayscale

Complete level

CT 2

DISCO

Language-based colorization

Automatic colorization

Inputs

Ours

To colorize images with descriptions of diverse levels of details, we propose a unified model that adaptively understands any-level descriptions and operates as follows: (i) For complete-level descriptions that cover major objects, the model precisely colorizes the image based on user requests (Fig. 1 first row, comprehensive descriptions for four cups of wines); (ii) for partial-level descriptions that focus on objects of interest, it colorizes unmentioned objects by cuing from image semantics when concentrating on their objects of interest (Fig. 1 second row, selective descriptions for only the jar and flowers); and (iii) for scarce-level descriptions that lack meaningful color information, it switches to the automatic colorization model (Fig. 1 third row, omitted descriptions for the pizza and restaurant).

To achieve the above goal, we present L-CAD to perform L anguage-based C olorization with A nylevel D escriptions. Given the inherent ambiguity in the number of objects mentioned in any-level descriptions, we leverage the pretrained cross-modality generative model ( i.e ., Stable Diffusion [34]) to utilize its robust language understanding for mentioned objects and rich color priors for unmentioned ones. However, as the generative model is not specifically designed for colorization, it faces challenges in spatial alignment with grayscale inputs. To address this, we develop a luminanceguided compression module that aligns with grayscale images in the pixel space, preserving local spatial structures. We also design a channel-extended convolution operator to prevent the ghosting effect by aligning with descriptions in the latent space. Additionally, to handle descriptions with diverse levels of details and complex scenarios, we develop an instance-aware sampling strategy that roughly estimates object contours in the latent space and assigns color features to their corresponding regions. Our approach is not limited by the number of objects mentioned in descriptions, enabling effective handling of any-level descriptions.

Our contributions could be summarized as follows:

We propose a unified model that adaptively understands any-level descriptions, achieving state-of-the-art performance in both automatic and language-based colorization methods.

We develop novel modules to preserve local spatial structures and prevent the ghosting effect by aligning with input conditions in both the pixel space and the latent space.

We present the instance-aware sampling strategy to correctly assign colors to corresponding objects, enabling effective colorization in diverse and complex scenarios.

2 Related Works

2.1 Language-based colorization

Language-based colorization aims to leverage flexible and user-friendly language descriptions to produce visually pleasing results with descriptions that contain objects and their corresponding colors. Manjunatha et al . [27] introduce the FILM pipeline, which employs feature-wise affine transformations to incorporate language descriptions in the grayscale image colorization process. Similarly, Chen et al . [7] utilize a recurrent attentive model to spatially fuse image and language features. To gain a deeper understanding of grayscale images, Xie et al . [49] use additional segmentation masks and introduce semantic segmentation as a side task. L-CoDe [45] and L-CoDer [5] rely on additionally annotated correspondences between object words and color words for guidance, enabling the model to assign specific color words to image regions. To achieve instance-aware colorization, L-CoIns [6] presents the grouping mechanism to automatically aggregate similar image patches. In this paper, we further explore the effective approach to enhance the robustness of language-based colorization, e.g ., assigning vivid colors for objects unmentioned in the description.

2.2 Automatic colorization

Automatic colorization aims to estimate the missing chromatic channels in grayscale images, generating diverse, colorful, and plausible results without relying on additional user-provided guidance. Chen et al . [8] pioneer the first deep-learning-based automatic colorization approach. With the advent of CNNs, which significantly increases the representation ability of the neural network, researchers have focused on developing adaptive feature extraction modules [10, 17, 23]. Recently, advanced generative models, e.g ., V AEs [9], GANs [4, 15, 40, 41], cINNs [2], and Transformers [18, 22, 47], have been employed to better fit real-world color distributions and produce vivid colorization results. Concurrently, other works have explored incorporating various priors to achieve accurate colorization semantics, including segmentation masks [56, 55], detection boxes [39], rich color prior [44, 52], and pretrained generative models [20, 46]. Despite these advances, automatic colorization still faces challenges in handling multi-modal uncertainty. Considering that rich image semantics are encapsulated in language descriptions, we utilize the prior knowledge from the pretrained cross-modality generative model, i.e ., Stable Diffusion [34], to alleviate this issue.

2.3 Text-driven image editing

Leveraging the flexibility of description representation, text-driven image editing methods enable precise control over image color, style, contour, and high-level semantics. Early works [24, 43, 51] develop text injection modules and design supervisory signals based on the conditional GAN [29]. Recently, with the development of diffusion models [13, 38], researchers show increasing interests in exploiting text-driven image generation [32, 33, 37], as well as diverse image editing tasks, e.g ., text-guided image inpainting [34, 48], inversion-based style transfer [54], and text-driven image-toimage translation [19]. To reduce resource consumption, some researchers build models based on pretrained generative models by advanced sampling strategies [28, 30] or finetuning their weights [35, 50]. Although language-based colorization could be considered as a specialized text-driven image editing task, existing methods utilizing pretrained generative models hardly or seldom succeed in instance-aware colorization due to their failure to spatially align with input grayscale images.

3 Methodology

In this section, we first provide a brief review of related diffusion models [13, 34, 38] in Sec. 3.1 to make this paper self-contained. Following that, we delve into the details of designed modules that spatially align with input conditions in Secs. 3.2 and 3.3. Next, we present the instance-aware sampling strategy for effective colorization in diverse and complex scenarios in Sec. 3.4. Finally, training details are shown in Sec. 3.5.

3.1 Preliminaries

Diffusion models [13, 38] are one type of generative models that learn to represent the distribution of images from a given dataset. During the inference process, these models iteratively refine a

𝓏0

CEC Block

CA Block

y

tex

𝓏𝑡

Diffusion

Process

𝓏𝑇

y

Partial-level

Description:

A girl is wearing a

purple t-shirt with the

looking at the camera.

Scarce-level

Description:

A colorful image.

Complete-level

Description:

Purple t-shirt,

a long brown table,

a green curtain,

the  pink cake in the

middle of the table.

(c) Any-level

Descriptions

Compression

Decoder

𝜖𝑡

Compression

Encoder

Colorized Image

Grayscale Image

Luminance

Encoder

Eq. (4)

Sampling Step

Eq. (6)

𝑀 est

Algorithm 1

Segmentation

Model

𝑡 = 0

𝑡 = 𝑡 - 1

Instance-aware

Sampling Strategy

𝓏0

random noise sample x T ∼ N (0 , 1) until it converges to a photorealistic image x 0 . At each step t ∈ { 0 , . . . , T } , the intermediate sample x t is computed as:

where ϵ t ∼ N (0 , 1) is the Gaussian noise at step t . In diffusion models, a neural network denoted as ϵ θ is employed at each step to predict the noise ϵ t , guiding the model in the process of reversing x t to x t -1 through a denoising procedure.

Diffusion models can be extended to handle conditional generative tasks by incorporating an additional condition y . In this case, the noise prediction network ϵ θ takes three parameters as input: the noise sample x t , the step t , and the condition y . This enables the diffusion process to generate results that are consistent with the given condition. To achieve the denoising objective, diffusion models typically utilize a simple L 2 loss, aiming to minimize the discrepancy between the learned image distribution and the target distribution. The loss function can be formulated as:

To address the challenges of resource consumption and high-resolution image generation, Stable Diffusion [34] introduces a perceptual compression model to make the latent space perceptually equivalent to the image space. This enables the diffusion process to be performed in the compressed latent space. Specifically, it adopts a compression encoder E to encode the given image x into latent space as z = E ( x ) , and a compression decoder D to reconstruct the image as ˜ x = D ( z ) . Notably, the noise prediction network ϵ θ is trained to learn the distribution of the latent representation z instead of the input image x .

In this paper, our goal is to leverage Stable Diffusion [34]'s robust language understanding and rich color priors for language-based colorization with any-level descriptions.

3.2 Luminance-guided image compression

While Stable Diffusion [34] demonstrates superior performance in text-driven image editing, it lacks the ability to preserve local spatial structures of input grayscale images, which is essential for the colorization task. To address this limitation, we propose the incorporation of an additional luminance

encoder in the pixel space. This encoder serves as a bridge between the colorization results and the grayscale images, ensuring better alignment between the two.

As shown in Fig. 2 (a), the colorful image is encoded into the latent representation by the compression encoder, while the luminance encoder extracts multi-scale features from grayscale images, preserving local structural semantics. These features are added directly into the corresponding scales of the compression decoder, guiding the decoding process from the latent space to the pixel space. The luminance encoder adopts the same architecture as the compression encoder, with the weights of the compression encoder and decoder fixed to retain prior knowledge from the pretrained model.

Inspired by LDL [25], we observe that erroneous pixels in regions with regular and sharp structures significantly damage visual perception. To address this, we intend to estimate an artifact map M art h,w , which would serve to indicate the probability of encountering artifacts at specific spatial position ( h, w ) within the colorized results. Specifically, we calculate the residual δ between the ground truth image x and the colorized result ˜ x . By computing the variance of the residual within local square windows at each position, we generate an artifact map denoted as M art as:

where p denotes the position index of the local window Ω centered on position ( h, w ) and N win is the local window size. Given that artifacts typically show up with high-frequency characteristics, areas with higher variances likely indicate where these artifacts are. We could apply the artifact map as a weight to the image reconstruction loss as L rec = ∥ M art ⊙ ( x -˜ x ) ∥ 1 .

Our total loss to train our model in pixel space is as follows:

where α = 1 . 0 and β = 0 . 5 are weighting parameters, the discriminator loss L dis and the perceptual loss L per are adopted to improve the overall quality of reconstructed images.

3.3 Semantic-aligned latent representation

In Stable Diffusion [34], descriptions are incorporated into the latent representation using CrossAttention (CA) blocks to ensure flexibility. However, this approach leads to the ghosting effect as color features are assigned outside the intended regions. To address this, we focus on maintaining semantic alignment between descriptions and grayscale images within the latent space.

In Fig. 2 (b), we replace Vanilla Convolution (VC) blocks in the noise prediction network with our Channel-Extended Convolution (CEC) blocks within each downsampling module. These CEC blocks receive the resized luminance features y lum from the luminance encoder in the pixel space as additional guidance. By leveraging extended channels, CEC blocks could effectively capture the local structural semantics of the luminance in the latent space. We formulate the channel-extended convolution as:

where h , w , p = h + i -N k / 2 and q = w + j -N k / 2 are position index of feature maps, f and f ′ are separately feature maps before and after performing the convolution, N k is the kernel size, N fix and N ext separately mean the number of fixed channels and extended channels, ¯ y lum denote resized luminance features. Mathematically, the CEC block is equivalent to using a stack of convolutions to extract features and add their output to the downsampling block. To preserve the robust language understanding and rich color priors of the pretrained generative model, we keep the weights ω fix fixed during training. Additionally, we initialize the weights of extended channels ω ext to zero, ensuring that our model maintains functional equivalence to the pretrained generative model prior to training.

We use skip connections between downsampling and upsampling modules to guide the upsampling process with luminance features. To expedite training convergence, we utilize Channel-Extended Convolution (CEC) blocks only in downsampling modules.

↓

The loss for training the denoising network in the latent space is:

where y tex is any-level descriptions encoded by the text encoder of CLIP [31].

3.4 Instance-aware sampling strategy

To ensure accurate color assignment to objects in images with diverse and complex descriptions, we propose an instance-aware sampling strategy. This strategy draws inspiration from the effectiveness of InstColor [39], which employs an auxiliary recognition model to enhance instance-aware performance. By incorporating the instance-aware sampling strategy into our colorization process, we aim to improve the accuracy and effectiveness of assigning colors to corresponding objects.

To improve instance-awareness, we employ a referring segmentation model (e.g., SAM [21]) to estimate object contours mentioned in the description ( M est ). Attention maps ( M att l ) at the l -th CA block control colorized regions for each color word. We normalize the attention maps with Sigmoid and refine them iteratively by aligning with downsampled estimated contours. After that, we apply the denoising process in DDIM [38]. Algorithm 1 outlines our instance-aware sampling strategy.

Algorithm 1: Instance-aware sampling strategy

input

:

Roughly estimated object contour

M

est

output :

Colorized latent representation

z

0

for

t

=

T . . .

1

do

_,

M

att

∗

=

ϵ

θ

(

z

t

, t, y

, y

tex

)

for

l

= 1

· · ·

L

do

ˆ

M

est

l

Downsampling(

M

est

, l

)

M←

M

att

l

)

ˆ

M

att

l

M

att

l

-

λ

∇

M

L

BCE

(

M

,

ˆ

M

est

l

)

end

ˆ

ϵ

t

,

_

=

ϵ

θ

(

z

t

, t, y

, y

tex

)

{

M

att

∗

ˆ

M

att

∗

}

z

t

-

1

= DDIM(

z

t

,

ˆ

ϵ

t

, t

)

end

3.5 Learning and implementation details

During training, we utilize the classifier-free guidance [14] to supervise the colorization process conditioned on descriptions. To increase robustness for any-level descriptions, we randomly replace 30% of the complete-level and partial-level descriptions with scarce-level descriptions. The pixel space modules are trained using Eq. (4) with a batch size of 8 for 10 epochs until convergence.

Our instance-aware sampling strategy does not heavily rely on precise segmentation results from the referring segmentation model. This is due to the following reasons: (i) Our sampling is performed in the latent space using low-resolution segmentation results, mitigating the impact of segmentation imprecision. (ii) When decoding the latent representation into the pixel space, the compression decoder, guided by local structural semantics from grayscale images, could adaptively fix color bleeding issues to produce visually pleasing results.

Grayscale

Description

ML2018

Ours

The two chairs on

the left are green,

two chairs on the

right are cyan, and

they are on a

brown carpet.

The man on the

left is wearing a

black

suit and red

tie, and the man

on the right is

wearing a red suit

and

black

tie.

L-CoDe

L-CoIns

Ground Truth

An orange cat is

sitting on a green

blanket.

A double decker

red white and

purple bus.

Complete level

Partial level

Grayscale

CIC

BigColor

Ours

DISCO

CT 2

Ground Truth

scarce level

Subsequently, we fix their weights and train the noise prediction network in the latent space using Eq. (6) for 50 epochs. The training process takes approximately 60 hours and is performed on 2 NVIDIA GeForce 3090Ti graphic cards. We employ the Adam optimizer with a learning rate of 1 × 10 -6 for the pixel space and 5 × 10 -5 for the latent space to minimize the losses. For sampling, we use the DDIM sampling method [38] with 50 sampling steps.

4 Experiment

Dataset. We conduct our experiments on language-based colorization datasets: (i) the extended COCO-Stuff dataset [45], which is built upon the COCO-Stuff dataset [3] by discarding unqualified samples for the colorization task, including 59K training images and 2.4K evaluation images; and (ii) the multi-instance dataset [6], which provides samples featuring multiple instances with different visual characteristics within a single image, including 65K training images and 7K evaluation images. For both datasets, each image is accompanied by a corresponding language description.

4.1 Comparison with state-of-the-art methods

We make comparisons with language-based colorization methods ( e.g ., LBIE [7], ML2018 [27], Xie2018 [49], L-CoDe [45], L-CoDer [5], and L-CoIns [6]) and automatic colorization methods

( e.g ., CIC [52], InstColor [39], ChromaGAN [40], BigColor [20], DISCO [47], and CT 2 [44]) to demonstrate that our model is capable of effectively handling any-level descriptions.

Qualitative comparisons. Previous language-based colorization methods assume that users provide comprehensive color descriptions, resulting in suboptimal performance, particularly for objects without corresponding color descriptions. In contrast, leveraging the prior knowledge of Stable Diffusion [34] and our newly proposed instance-aware sampling strategy, our L-CAD presents vivid colorization results with any-level descriptions, as shown in Fig. 3. we further make comparisons with automatic colorization methods using scarce-level descriptions that lack meaningful color information. As shown in Fig. 4, our model still produces superior colorization results.

Quantitative comparisons. Following previous works [5, 6, 45], we use Peak Signal-to-Noise Ratio (PSNR) [16], Structural Similarity Index Measure (SSIM) [42], and Learned Perceptual Image Patch Similarity (LPIPS) [53] to qualitatively evaluate the quality of colorization. With complete-level and partial-level descriptions, our method (L-CAD) outperforms all language-based colorization methods. Furthermore, with scarce-level descriptions, our method shows a distinct advantage over automatic colorization methods. As shown in Tab. 1, our method achieves the best PSNR, SSIM, and LPIPS scores on both datasets.

4.2 User study

We further conduct user studies to evaluate the subjective perception of human observers: (i) Corresponding experiment: We apply language-based colorization methods to colorize images with random complete-level or partial-level descriptions. Participants are instructed to select the image that best matches the description. (ii) Reality experiment: Participants are asked to select the image they perceive to be the most visually realistic among the real image, images colorized by our method with scarce-level descriptions, and those colorized by automatic colorization methods. For each experiment, we randomly select 100 samples from the testing images of each dataset, and conduct them independently by 25 volunteers on the Amazon Mechanical Turk (AMT). As shown in Tab. 2, our method achieves the highest scores in both experiments.

4.3 Ablation study and discussion

We create three baselines to study the impact of our proposed modules and the sampling strategy. The evaluation scores and colorization results of the ablation study are shown in Tab. 1 and Fig. 5.

W/o Luminance-guided Image Compression (LIC). We disable the luminance encoder in the pixel space and revert to using the perceptual compression model in [34]. Without the guidance of multi-scale grayscale features, the model cannot correctly preserve local spatial structures.

W/o Semantic-aligned Latent Representation (SLR). We remove luminance features that guide the semantic alignment in the latent space and therefore replace channel-extended convolution blocks with vanilla convolution blocks. As a result, the obvious ghosting effect occurs in colorized images.

Grayscale

Description

W/o

LIC

Ours

The woman on

the left wears in

pink , the

woman on the

right wears in

dark blue.

W/o

SLR

W/o

ISS

Ground Truth

Yellow shirted

tennis player

looking for

incoming ball.

A colorful image.

Complete level

Partial level

Scarce level

0.17

0.12

0.12

LEX

LEX

LEX

LEX

Grayscale

Description

ControlNet

Ours

Pix2PixZero

SDEdit

Ground Truth

There is a

yellow-green

tennis ball on

the dark blue

tennis racket.

A blue sign

with a white

light on top.

Partial level

Complete level

eecetto

W/o Instance-aware Sampling Strategy (ISS). We replace the instance-aware sampling strategy with the standard DDIM [38], which significantly degrades the performance of the model to correctly assign colors to corresponding objects that have different descriptions.

Discussion. Recent studies ( e.g ., ControlNet [50], Pix2PixZero [30], and SDEdit [28]) could also edit images with descriptions, based on the pretrained generative models. However, these methods are not tailored to the colorization task, resulting in challenges related to preserving local spatial structures, leveraging rich color priors, and learning correspondence between objects and color words. These limitations make it difficult for them to perform instance-aware colorization with any-level descriptions. We present their unsuitability in Fig. 6.

4.4 Evaluation on ImageNet dataset

We conduct extensive experiments on the widely-used ImageNet dataset [36], including 1.3M images covering 1000 categories, to evaluate the performance of our L-CAD for automatic colorization. To provide a comprehensive evaluation, we introduce the Fréchet inception distance [12] to quantify the distribution gap between colorized images and ground truth images, as well as the colorfulness score [11] to reflect the vividness of colorized images. Following previous works [2, 22, 44], we evaluate our model on the first 5K images of the public validation set, where all the test images are center cropped and resized into 256 × 256 resolution.

As shown in Tab. 3, our model achieves top scores in FID, PSNR, SSIM, and LPIPS metrics, and demonstrates comparable performance in colorfulness metrics. These results indicate that our model aims to achieve photo-realistic colorization, emphasizing natural tones rather than exaggerated saturation. It should be noted that due to the absence of provided descriptions in the ImageNet

The two men wear

khaki and blue shirts,

and the curtains

behind them were

also golden.

1940. "Japanese fruit farmer and his son. Placer County, California."

1906.  'Scene at Rock

Creek, zoo park,

Washington, D.C."

A colorful image.

There's a black car on the left and a cyan car on the right.

There's a blue car on the left and a red car on the right.

1913.  "Main Street, a

commercial and

residential block at the

corner of Main and

Cabot."

BIJOU

BIJOU

BIJOU

dataset [36], our model is trained solely with scarce-level descriptions ( e.g ., 'a colorful image') and leverage the rich image semantics encapsulated in language-based descriptions by initializing weights pretrained on the extended COCO-Stuff dataset [45]. This deviates from the training strategy adopted in language-based colorization datasets [6, 45] (detailed in Sec. 3.5), potentially affecting the performance of our model.

4.5 Application

We apply our method to colorize legacy black-and-white photos using descriptions at the complete, partial, and scarce levels, respectively. Results are presented sequentially from left to right in Fig. 7, which demonstrates the robust generalization capability of our model.

5 Conclusion

We present L-CAD , a unified model to perform L anguage-based C olorization with A ny-level D escriptions. Leveraging the prior knowledge of the pretrained model, we design novel modules to preserve local spatial structures and prevent the ghosting effect. we further propose the instanceaware sampling strategy to colorize diverse and complex scenarios. We demonstrate our model could handle any-level descriptions by making comparisons with language-based colorization methods with complete-level and partial-level descriptions, as well as automatic colorization methods with scarelevel descriptions. Both qualitative and quantitative results demonstrate our superior performance.

Limitations. Our model as a diffusion model takes an additional sampling process for visualizing colorization results. This requirement considerably slows down the generation speed, and further limits the potential for real-time applications of our model. To address this concern, adopting advanced fast sampling methods ( e.g ., DPM-Solver [26]) could be a promising direction.

Acknowledgement. This work is supported by the National Key R&D Program of China under Grant No. 2021ZD0109800, the National Natural Science Foundation of China under Grand No. 62136001, 62088102, and Program for Youth Innovative Research Team of BUPT No. 2023QNTD02. Yu Li is supported by Shenzhen Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone, under Grant No. HTHZQSWS-KCCYB-2023052.

References

J. Antic. A deep learning based project for colorizing and restoring old images (and video!).

L. Ardizzone, C. Lüth, J. Kruse, C. Rother, and U. Köthe. Guided image generation with conditional invertible neural networks. arXiv preprint arXiv:1907.02392 , 2019.

H. Caesar, J. Uijlings, and V. Ferrari. COCO-Stuff: Thing and stuff classes in context. In CVPR , 2018.

Y. Cao, Z. Zhou, W. Zhang, and Y. Yu. Unsupervised diverse colorization via generative adversarial networks. In ECML-PKDD , 2017.

Z. Chang, S. Weng, Y. Li, S. Li, and B. Shi. L-CoDer: Language-based colorization with color-object decoupling transformer. In ECCV , 2022.

Z. Chang, S. Weng, P. Zhang, Y. Li, S. Li, and B. Shi. L-CoIns: Language-based colorization with instance awareness. In CVPR , 2023.

J. Chen, Y. Shen, J. Gao, J. Liu, and X. Liu. Language-based image editing with recurrent attentive models. In CVPR , 2018.

Z. Cheng, Q. Yang, and B. Sheng. Deep colorization. In ICCV , 2015.

A. Deshpande, J. Lu, M.-C. Yeh, M. Jin Chong, and D. Forsyth. Learning diverse image colorization. In CVPR , 2017.

S. Guadarrama, R. Dahl, D. Bieber, M. Norouzi, J. Shlens, and K. Murphy. PixColor: Pixel recursive colorization. arXiv preprint arXiv:1705.07208 , 2017.

D. Hasler and S. E. Suesstrunk. Measuring colorfulness in natural images. In Human vision and electronic imaging VIII , 2003.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter. GANs trained by a two time-scale update rule converge to a nash equilibrium. In NIPS , 2017.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NIPS , 2020.

J. Ho and T. Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications .

Z. Huang, N. Zhao, and J. Liao. UniColor: A unified framework for multi-modal colorization with transformer. In SIGGRAPH Asia , 2022.

Q. Huynh-Thu and M. Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics letters , 2008.

S. Iizuka, E. Simo-Serra, and H. Ishikawa. Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ToG , 2016.

X. Ji, B. Jiang, D. Luo, G. Tao, W. Chu, Z. Xie, C. Wang, and Y. Tai. ColorFormer: Image colorization via color memory assisted hybrid-attention transformer. In ECCV , 2022.

B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani. Imagic: Text-based real image editing with diffusion models. In CVPR , 2023.

G. Kim, K. Kang, S. Kim, H. Lee, S. Kim, J. Kim, S.-H. Baek, and S. Cho. BigColor: Colorization using a generative color prior for natural images. In ECCV , 2022.

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

M. Kumar, D. Weissenborn, and N. Kalchbrenner. Colorization transformer. In ICLR , 2021.

G. Larsson, M. Maire, and G. Shakhnarovich. Learning representations for automatic colorization. In ECCV , 2016.

B. Li, X. Qi, T. Lukasiewicz, and P. H. Torr. ManiGAN: Text-guided image manipulation. In CVPR , 2020.

J. Liang, H. Zeng, and L. Zhang. Details or artifacts: A locally discriminative learning approach to realistic image super-resolution. In CVPR , 2022.

C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NIPS , 2022.

V. Manjunatha, M. Iyyer, J. Boyd-Graber, and L. Davis. Learning to color from language. In NAACL , 2018.

C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR , 2021.

T. Miyato and M. Koyama. CGANs with projection discriminator. In ICLR , 2018.

G. Parmar, K. K. Singh, R. Zhang, Y. Li, J. Lu, and J.-Y. Zhu. Zero-shot image-to-image translation. arXiv preprint arXiv:2302.03027 , 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In ICML , 2021.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR , 2023.

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV , 2015.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NIPS , 2022.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In ICLR , 2021.

J.-W. Su, H.-K. Chu, and J.-B. Huang. Instance-aware image colorization. In CVPR , 2020.

P. Vitoria, L. Raad, and C. Ballester. ChromaGAN: Adversarial picture colorization with semantic class distribution. In WACV , 2020.

Y. Wang, M. Xia, L. Qi, J. Shao, and Y. Qiao. PalGAN: Image colorization with palette generative adversarial networks. In ECCV , 2022.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP , 2004.

S. Weng, W. Li, D. Li, H. Jin, and B. Shi. Conditional image repainting via semantic bridge and piecewise value function. In ECCV , 2020.

S. Weng, J. Sun, Y. Li, S. Li, and B. Shi. CT 2 : Colorization transformer via color tokens. In ECCV , 2022.

S. Weng, H. Wu, Z. C. Chang, J. Tang, S. Li, and B. Shi. L-CoDe: Language-based colorization using color-object decoupled conditions. In AAAI , 2022.

Y. Wu, X. Wang, Y. Li, H. Zhang, X. Zhao, and Y. Shan. Towards vivid and diverse image colorization with generative color prior. In ICCV , 2021.

M. Xia, W. Hu, T.-T. Wong, and J. Wang. Disentangled image colorization via global anchors. TOG , 2022.

S. Xie, Z. Zhang, Z. Lin, T. Hinz, and K. Zhang. SmartBrush: Text and shape guided object inpainting with diffusion model. In CVPR , 2023.

Y. Xie. Language-guided image colorization. Master's thesis, ETH Zurich, Departement of Computer Science, 2018.

L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models, 2023.

L. Zhang, Q. Chen, B. Hu, and S. Jiang. Text-guided neural image inpainting. In ACM MM , 2020.

R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In ECCV , 2016.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

Y. Zhang, N. Huang, F. Tang, H. Huang, C. Ma, W. Dong, and C. Xu. Inversion-based creativity transfer with diffusion models. In CVPR , 2023.

J. Zhao, J. Han, L. Shao, and C. G. Snoek. Pixelated semantic colorization. IJCV , 2020.

J. Zhao, L. Liu, C. G. Snoek, J. Han, and L. Shao. Pixel-level semantics guided image colorization. In BMVC , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a model to adapt pretrained Stable Diffusion for language-conditioned colorization. In specific, they propose a luminance encoder that produces latents $y_lum$. This conditions the compression decoder and the denoiser of the diffusion model. The conditioning of the denoiser is done using a \"Channel Extended Convolution block\". Further during sampling, they utilize gradients from a semantic segmentation model to help with instance differentiation.\n\nComparisons are made to several language-conditioned and unconditional baselines across 2 datasets: extended coco-stuff and a (private) multi-instance dataset. They outperform these baselines on both quantitative metrics and mechanical turk. They show qualitative samples where they are able to colorize multi-level descriptions better than the baselines.\n\nSTRENGTHS:\nThe results shown by their proposed approach are quite strong. The authors propose to augment a pretrained stable diffusion model to achieve language-conditoned colorizaiion which is novel and interesting. The authors provide adequate background information.\n\nWEAKNESSES:\nWhile the model works well, imho some novelty aspects are overclaimed and some simpler baselines are not ablated. I initially rate the paper slightly below borderline but I'm happy to update my rating it the authors address my concerns. See below for a detailed list of questions.\n\nQUESTIONS:\n**Most Important**: Amongst the standard training hyperparameters, the paper introduces $\\lambda$ for sampling, the loss hyperparameters $\\alpha$ and $\\beta$ and the number of elements $N_{win}^2$ in the window for the loss in Eq 3). How are these hyperparameters tuned? \n\n\nLuminance-guided image compression\n-----------------------------------------\n\n* The design of conditioning the “compression decoder” with the grayscale features makes sense. To be more self-conditioned, I suggest the authors to a) Briefly explain the architecture of the compression encoder + decoder b) Describe how exactly the conditioning from the luminance encoder features is incorporated in the compression decoder?\n* In Section 3.2, the authors introduce a discriminator loss but this is not explained or ablated. Is this introduced in this work are part of stable diffusion?\n* The authors introduce a loss that promotes smoothness by upweighting windows with high variance (Eq 3). The authors should ablate this weighting factor. Moreover, I think only the “luminance conditioning” is introduced in this work. So I suggest that the authors move the part that focuses on the training the compression encoder+decoder with the loss functions into the preliminaries and focus only on the luminance conditioning in this subsection.\n\nSemantic-aligned latent representation\n-------------------------\n\n* Imho, the novelty of this section is overclaimed. The “semantic-aligned” latent representation amounts to a zero-initialized convolution which takes as input to the grayscale features and is added to the output of denoising downsampling blocks. It is great that it works, but the authors should describe it as it is and not overclaim novelty.\n* Have the authors tried the obvious baseline that just concatenates $y_lum$ with $z_t$ at the inputs with downsampling to match the shapes?\n\nInstance-aware sampling strategy\n-----------------------\n\n* IIUC, the goal is to differentiate between different instances and if the output of a semantic segmentation model is used to provide this information as some sort of guidance or ground truth. If this is indeed the case, then the authors should ablate the simpler baseline, which is just to condition the denoising process with the output of a semantic segmentation model during training (as they do with the grayscale images)\n* What eactly is the semantic segmentation model used here? The authors say (eg SAM but it it not clear if SAM is used)\n* The authors should mention the shapes of the $M_est$ and $M_att$ and how they are matched.\n* What is the role of the softmax? Since it is cross-attention with the text tokens, aren’t the attention values normalized across the #text tokens already?\n\nResults\n---------\n* Can the authors confirm that the images apart from the 2.9K descriptions that are removed, all have color information?\n* The authors describe a multi-instance dataset. Are there plans to opensource this dataset?\n* In Figure 1, the authors describe three levels of colorization descriptions. The authors show some qualitiative results per-level, can they report some quantitative results as well?\n* Can the authors also report some of their failure cases, if any?\n\nOther\n-------\nFigure 2 combines both the training and sampling loop of the diffusion process, so it is quite confusing. I suggest that the authors separate both and clearly label the diagram wherever the loss is applicable.\n\nLIMITATIONS:\nI do not believe there is negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a modification over Stable Diffusion to adapt it to perform language-based colorization of images with varying levels of description details.\nThe method is based on three modifications. First, in addition to the auto-encoder used for SD, the authors add another encoder that is tasked with preserving the spatial features of the input image. Second, the convolutions in the downsampling layers of the UNet are replaced with a novel Channel-Extended Convolution to encourage the reliance on the encoded spatial features. Finally, a segmentation model is employed to encourage correct object-level color assignment.\nExtensive experiments are conducted to demonstrate the method's superiority over both language-conditioned colorization methods and automatic colorization methods.\n\nSTRENGTHS:\n- The authors conduct very extensive experiments, using various datasets and human evaluation.\n- Overall, the reviewer found the qualitative results to be convincing.\n- The idea of slightly modifying the Stable Diffusion architecture for other tasks is interesting and can possibly be generalized to other tasks.\n\nWEAKNESSES:\nPlease note that the low confidence is due to the fact that the reviewer is not familiar with literature on image colorization, and therefore feels less confident in giving an assessment of this work. \n\nReadability:\nOverall, the reviewer found the method section to be a bit hard to follow. Intuitions are lacking in some parts, and familiarity with previous works is required to follow the explanations.\n- The writing of the preliminaries section on diffusion models is a bit confusing and inaccurate. For example, it starts with describing the inference (L. 106-107) but then turns to describe the training process (Eq. 1 and on).  \n- Section 3.3 (Channel-Extended Convolution) is not entirely clear. A supporting figure demonstrating the VC vs. CEC would be beneficial to better understand the module, its motivation and novelty.\n- Section 3.4 heavily relies on familiarity with existing colorization methods, and the manipulation to the cross-attention maps is not clear to the reviewer (e.g., why employ a sigmoid non-linearity on the attention?) \n\nEvaluation:\n- The authors claim that when ablating the Instance-aware Sampling Strategy (ISS) \"significantly degrades the performance of the model to correctly assign colors to corresponding objects that have different descriptions.\" (L.256) however, the results in Fig. 5 and Tab. 1 appear to be pretty similar to the proposed method.\n- The comparison to image editing methods are partial, however given the extensive comparisons, this is not a major concern.\n\nQUESTIONS:\n- The authors mention the \"ghosting effect\" several times, but no explanation is given as to what it is (e.g., L. 153).\n- Why is the CEC block only used in the first half of the UNet (downsampling)?\n- How robust is the method to OOD outputs? For example, what happens if you train on COCO-Stuff and evaluate on multi-instance?\n\nLIMITATIONS:\nLimitations are discussed briefly. The reviewer thinks that a more in-depth comparison of runtime to SD and to the baselines is needed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a novel approach to image colorization, which demonstrates superior performance in language-based colorization methods. The proposed model employs language descriptions at varying detail levels to produce high-quality, customizable colorized images by diffusion in the latent space. A key innovation is L-CAD's adaptive understanding of any-level descriptions, facilitating precise colorization based on user requests. To ensure proper spatial alignment with grayscale inputs and avoid ghosting effects, a luminance-guided compression module, and a channel-extended convolution operator are introduced. An instance-aware sampling strategy was adopted from previous literature to enhance color assignment to objects. L-CAD exhibits very good results in both quantitative and qualitatve experiments.

STRENGTHS:
- The paper is well-written and easy to follow. 
- The problem that they tackle is interesting and valuable. 
- The results are promising and the ablation studies to disentangle different modules have been conducted.

WEAKNESSES:
- The method is a little complex and needs multiple modules to work properly.

QUESTIONS:
- Can the authors please compare training and inference time between their method and the baselines?

LIMITATIONS:
- The authors have mentioned that their model is relatively slow.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper presents a well engineered method for Stable
Diffusion-based text-guided image colorization. The paper addresses
the challenge of structural fidelity to the original grayscale image
by additional conditioning modules in the denoiser architecture. The
paper also proposes an instance-aware sampling procedure that uses
object segmentations to encourage accurate color assignment. Extensive
quantitative and qualitative comparison show the proposed method
outperforms the compared baselines.

STRENGTHS:
- This is a well engineered method. The design decisions appear
  reasonable and are justified with ablation studies.
- The paper is well written, the ideas are clear and easy to understand.
- The qualitative results are remarkable and quantitative comparisons
  show the advantage of the proposed method in standard benchmarks and
  user preference studies.

WEAKNESSES:
- The colorization baselines in Figure 1 and 4 look much weaker than
  in the original papers (e.g. [43] Fig. 6 and [40] Fig. 5). I wonder
  how the proposed method would compare in the referenced figures.
- It would be interesting to include in the quantitative comparison
  the number of sampling steps required for each method, as this is a
  disadvantage of the diffusion-based methods compared to the GAN-based
  ones.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of colorizing images with descriptions of diverse levels of details. The key idea of the work is to propose a unified model that adaptively understands any-level descriptions by leveraging the pretrained cross-modality generative model. Additionally, the paper introduces modules that aid in preserving local spatial structures and prevent the ghosting effect by aligning with input conditions in both the pixel space and the latent space. Further, the paper presents an instance-aware sampling strategy to correctly assign colors to corresponding objects, enabling effective colorization in diverse and complex scenarios. The work demonstrates state-of-the-art performance in both automatic and language-based colorization methods.\n\nSTRENGTHS:\nThe paper is well-written and well-motivated. Qualitative and quantitative comparisons to existing methods show the effectiveness of the proposed approach in language based colorization and automatic colorization. User studies and ablation studies sufficiently justify the key conclusions made in the paper.\n\nWEAKNESSES:\nOne of the key weaknesses of the paper is the readability of a few sub-sections. For example, I struggle to understand the detailed implementation/intuition for section 3.2 where the paper introduces luminance guided image compression and how it helps preserve local structural semantics. It is not very intuitive as to how it can help bridge alignment between colorization results and grayscale images. Some more detailed intuitions could help readers to better follow this section. Similarly the justifications for equation 3 could be further elaborated. In lines 153-154, on the ghosting effects, I am curious to know if this is still the case with using region based guidance (for example, see below two references). I will revise my scores based on the clarifications in the rebuttal.\n\nYang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., ... & Wang, L. (2023). Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14246-14255).\n\nLi, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., ... & Lee, Y. J. (2023). Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22511-22521).\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nAuthors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}