L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors
Authors: Zheng Chang
Shuchen Weng
Peixuan Zhang
Yu Li
Si Li
Boxin Shi
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Colorization
Language-based generation
Diffusion model

Abstract: Language-based colorization produces plausible and visually pleasing colors under the guidance of user-friendly natural language descriptions. Previous methods implicitly assume that users provide comprehensive color descriptions for most of the objects in the image, which leads to suboptimal performance. In this paper, we propose a unified model to perform language-based colorization with any-level descriptions. We leverage the pretrained cross-modality generative model for its robust language understanding and rich color priors to handle the inherent ambiguity of any-level descriptions. We further design modules to align with input conditions to preserve local spatial structures and prevent the ghosting effect. With the proposed novel sampling strategy, our model achieves instance-aware colorization in diverse and complex scenarios. Extensive experimental results demonstrate our advantages of effectively handling any-level descriptions and outperforming both language-based and automatic colorization methods. The code and pretrained models
are available at: https://github.com/changzheng123/L-CAD.

L-CAD: Language-based Colorization with Any-level Descriptions using Diffusion Priors
Zheng Chang*
Shuchen #2,
Peixuan Zhang' Li Li Boxin Shi
1 School of Artificial Intelligence, Beijing University of Posts and Telecommunications
National Key Laboratory for Multimedia Information Processing
School of Computer Science, Peking University
3 National Engineering Research Center of Visual Technology
School of Computer Science, Peking University
International Digital Economy Academy
(zhengchang98, pxw.aannnn...xpx.nn..mm lisii)bbpt.eddeeeeee pebbupt.edd.... cn
(shuchenweng, shiboxin]0pku. edu.cr liyudidea, edu.
Abstract
Language-based colorization produces plausible and visually pleasing colors under the guidance of user-friendly natural language descriptions. Previous methods implicitly assume that users provide comprehensive color descriptions for most of the objects in the image, which leads to suboptimal performance. In this paper, we propose unified model to perform language-bass based colorization with any- level descriptions. We leverage the pretrained cross- cooss-oollltyy generative model for its robust language understanding and rich color priors to handle the inherent ambiguity of any-level descriptions. We further design modules to align with input conditions to preserve local spatial structures and prevent the ghosting effect. With the proposed novel sampling strategy, our model achieves instance-awar colorization in diverse and complex scenarios. Extensive experimental results demonstrate our advantages of effectively handling any-level descriptions and outperforming both language-based and automatic colorization methods. The code and pretrained models are available at: bttp::/gtthubb.oommcnnnggee////
Introduction
Image colorization is a challenging task that involves converting grayscale images into plausible and visually pleasing colorful ones. Language-baser colorization approaches (e.g.. [5] 6 277 use natural language descriptions guidance to produce more controllable colorized images. These methods cater users' specific requests, enabling them to provide more specific and nuanced color preferences. As automatic colorization (e.g., [39 44 (52) often encounters color ambiguity for common objects (e.g., flower colors), language-basee colorization has shown promising results in producing high-quality and customizable colorized images with user-friendly input.
While language-bass based colorization methods have improved the consistency between language descrip- tions and colorization results through feature fusion 77 (2)), decoupling color-object space [5] 45). and aggregating similar patches (6), they implicitly assume that users provide comprehensive color descriptions for most of the objects in the image. This assumption often leads to suboptimal per- formance, especially for objects without corresponding color descriptions. Furthermore, we have observed that users commonly assign colors only to their objects of interest.
Equal contribution. Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The colors the wine are red, blue, vello,, and cyan from left to fight
Scarce Partial Complete
Grasscllee
4.44
purple meson next pilk flowers.
Graiscale
LiCoee
A colorful Image.
Grayocale
L-CoDe
L.Cons
DSCC
CT
L-CAD
Inputs
Language-basee colorization
Automatic colorization
Ours
Figure I: Colorization with any-level descriptions, comparing uinguage-based colorization methods (e.g.. L-CoDe [45] and L-Colns [6]) and automatic colorization methods (e.g., DISCO [47] and CT
44 First row: For complete-level descriptions, automatic colorization methods is unable
to colorize images as requested by the user. Second row: For partial-level descriptions, automatic colorization methods fail to colorize mentioned objects with their corresponding colors, while language-basee colorization methods struggle to present appropriate colors for unmentioned objects. Third row: For scarce-level descriptions, language- beseed colorization methods show suboptimal performance without meaningful color hints. In contrast, our method able to colorize images with any-level descriptions.
To colorize images with descriptions of diverse levels of details, we propose unified model that adaptively understands any-level descriptions and operates as follows: (i) For complete- descriptions that cover major objects, the model precisely colorizes the image based (Fig. first row, comprehensive descriptions for four cups of wines); (ii) for partial-level descriptions that focus objects of interest, it colorizes unmentioned objects by cuing from image semantics when concentrating on their objects of interest (Fig, second row, selective descriptions for only the jar and flowers); and (iii) for scarce-level descriptions that lack meaningful color information, switches to the automatic colorization model (Fig. third row, omitted descriptions for the pizza and restaurant).
requests
To achieve the above goal, we present L-CAD to perform Language-bast Colorization with Any- level Descriptions. Given the inherent ambiguity in the number of objects mentioned in lnvel descriptions, we leverage the pretrained cross-modality generative model (i.e., Stable Diffusion 34)) to utilize its robust language understanding for mentioned objects and rich color priors for unmentioned ones. However, as the generative model is not specifically designed for colorization, faces challenges in spatial alignment with grayscale inputs. To address this, we develop luminance- guided compression module that aligns with grayscale images in the pixel space, preserving local spatial structures. We also design channel-extended convolution operator to prevent the ghosting effect by aligning with descriptions in the latent space. Additionally, to handle descriptions with develop an instance-aware sampling strategy that
diverse levels of details and complex scenarios, roughly estimates object contours in the latent space and assigns color features to their corresponding regions. Our approach not limited by the number of objects mentioned in descriptions, enabling effective handling of any-level descriptions.
Our contributions could be summarized follows:
We propose a unified model that adaptively understands any-level descriptions, achieving ofate-of-the-an performance in both automatic and language- based colorization methods.
We develop novel modules to preserve local spatial structures and prevent the ghosting effect by aligning with input conditions in both the pixel space and the latent space.
We present the instance- aware sampling strategy correctly assign colors to corresponding objects, enabling effective colorization in diverse and complex scenarios.

---

2 Related Works
2.1 Language-based colorization
Language-basee colorization aims to leverage flexible and user suer-friendly language descriptions to produce visually pleasing results with descriptions that contain objects and their corresponding colors. Manjunatha et al. [27] introduce the FILM pipeline, which employs feature-wis wise affine transforma- tions to incorporate language descriptions in the grayscale image colorization process. Similarly, Chen et al. 7 utilize a recurrent attentive model to spatially fuse image and language features. To gain a deeper understanding of grayscale images, Xie al. [49 use additional segmentation masks and introduce semantic segmentation as side task. L-CoDe 45 and L-CoDer [5] rely on additionally annotated correspondences between object words and color words for guidance, enabling the model to assign specific color words to image regions. To achieve instance-aware colorization, L-CoIns [6] presents the grouping mechanism to automatically aggregate similar image patches. In this paper, we further explore the effective approach to enhance the robustness of language- baaed colorization, e.g., assigning vivid colors for objects unmentioned in the description.
2.2 Automatic colorization
Automatic colorization aims to estimate the missing chromatic channels in grayscale images, gener- ating diverse, colorful, and plausible results without relying on additional user-provided guidance. Chen et al. B pioneer the first deep-learning- automatic colorization approach. With the advent of CNNs, which significantly increases the representation ability of the neural network, researchers have focused on developing adaptive feature extraction modules [TO 17 23]. Recently, vdncee generative models, e.g., VAEs (9]. GANs [5] 40 41], cINNs (2). and Transformers [18] 22 47),
have been employed to better fit real-world color distributions and produce vivid colorization results. Concurrently, other works have explored incorporating various priors to achieve accurate colorization semantics, including segmentation masks [56 55), detection boxes [39], rich color prior 44 [2].
and pretrained generative models 20 46). Despite these advances, automatic colorization still faces challenges in handling multi-modal uncertainty. Considering that rich image semantics are encapsu- lated in language descriptions, utilize the prior knowledge from the pretrained cross-modality generative model, i.e., Stable Diffusion
34 to alleviate this issue.
2.3 Text-driven image editing
Leveraging the flexibility of description representation, text-driven image editing methods enable precise control over image color, style, contour, and high-level semanties. Early works 24
43
develop text injection modules and design supervisory signals based on the conditional GAN 22
Recently, with the development of diffusion models [
38 researchers show increasing interests as well as diverse image editing tasks, egg., text-guided image inpainting (34 [88, inversion style transfer (54), and text-driven image-to-
in exploiting text-driven image generation
32
33
37
image translation [19]. To reduce resource consumption, some researchers build models based on pretrained generative models by advanced sampling strategies (28] 30 or finetuning their weights
[35] 50]. Although language-based colorization could be considered as specialized text-driven image editing task, existing methods utilizing pretrained generative models hardly or seldom succeed in instance-aware colorization due to their failure to spatially align with input grayscale images.
3 Methodology
In this section, we first provide brief review of related diffusion models 13 34 38 in Sec. 3.1 to make this paper self-containel Following that, we delve into the details of designed modules that spatially align with input conditions in Secs. 3.2 and 3.3 Next, we present the instance-aware sampling strategy for effective colorization in diverse and complex scenarios in Sec. 3.4 Finally,
training details are shown Sec. 3.5
3.1 Preliminaries
Diffusion models 13 are one type of generative models that learn to represent the distribution of images from a given dataset. During the inference process, these models iteratively refine
3

---

(a) Pixel Space
(b) Latent Space
(c) Any-level Descriptions
Comprestion
Exccaa
P
Noise Predic tion Network
g
Scarcellevel
Description:
A colorfulimage
noLtimage
Partaa-ievel
LES.
Description:
is wearing
Lunirree Encok
looking the camera.
CEC Block VC Block
CA Block
Grapsc le Image
Completa-level Description:
Sampling Step
Comporsion Decette
Wgortinn
Purple s-shift, long brown table, green curtain, the pinki cake in the middle of the lable.
iistance-aaarte
Sampling Stratogy
Segmentation Model
Colorbed
Figure 2: The framework of L-CAD. (a) We introduce an additional luminance encoder to capture multi-scale features from the grayscale image, which preserves local spatial structures of grayscale images and guides the decoding process from the latent space back to pixel space (Sec. (.2). (b) We modify the noise prediction network by replacing Vanilla Convolution (VC) blocks with proposed Channel-Extended Convolution (CEC) blocks within each downsampling module before Cross-Attention (CA) blocks. This prevents the ghosting effect by aligning with descriptions in the latent space (Sec. 3.3). For visualizing the colorized image, we further present the instance-aware sampling strategy correctly assign colors corresponding objects for descriptions with diverse levels of details and complex scenarios (Sec. 3.4). (c) During training, our model randomly receives any-level descriptions increase the robustness of colorization (Sec. 3.5
random noise sample sy (O, 1) until it converges photorealistic image To- At each step e [o,, the intermediate sample x, is computed as:
24 /o + vI accs
(1)
where &, (0, is the Gaussian noise step t. In diffusion models, neural network denoted as eg is employed at each step to predict the noise et, guiding the model the process of reversing xy to Tt-1 through a denoising procedure.
Diffusion models can be extended handle conditional generative tasks by incorporating an additional condition y. In this case, the noise prediction network takes three parameters as input: the noise sample 3t, the step 4, and the condition z/. This enables the diffusion process to generate results that are consistent with the given condition. To achieve the denoising objective, diffusson models typically utilize a simple L2 loss, aiming to minimize the discrepancy between the learned image distribution and the target distribution. The loss function can be formulated as:
aa R0, [lke - co(x,, 1,,)
To address the challenges of resource consumption and high-resolution image generation, Stable Diffusion (34] introduces perceptual compression model make the latent space perceptually equivalent to the image space. This enables the diffusion process to be performed in the compressed latent space. Specifically, it adopts compression encoder & to encode the given image into latent space E(x), (z), and compression decoder D to reconstruct the image as ă = D(z). Notably, the noise prediction network eg is trained to learn the distribution of the latent representation instead of the input image
32 's robust language understanding and rich In this paper, goal is to leverage Stable Diffusion color priors for language- based colorization with anv-level descriptions.
3.2 Luminance-guided image compression
While Stable Diffusion 34 demonstrates superior performance in text-driven image editing, it lacks the ability to preserve local spatial structures of input grayscale images, which essential for the colorization task. To address this limitation, propose the incorporation of additional luminance

---

encoder in the pixel space. This encoder serves bridge between the colorization results and the grayscale images, ensuring better alignment between the two.
As shown in Fig. (a), the colorful image is encoded into the latent representation by the compression encoder, while the luminance encoder extracts multi-scale features from grayscale images, preserving local structural semantics. These features are added directly into the corresponding scales of the compression decoder, guiding the decoding process from the latent space the pixel space. The luminance encoder adopts the same architecture as the compression encoder, with the weights of the compression encoder and decoder fixed retain prior knowledge from the pretrained model.
Inspired by LDL [25] we observe that erroneous pixels in regions with regular and sharp structures significantly damage visual perception. To address this, we intend to estimate an artifact map Mr which would serve to indicate the probability of encountering artifacts at specific spatial position (h, w) within the colorized results. Specifically, we calculate the residual ò between the ground truth image and the colorized result ž. By computing the variance of the residual within local square windows at each position, we generate an artifact map denoted as M art as:
M
Σ
8p ( Wwn
Hp
Up
6p N2
(3)
pE
PE(R
win
where p denotes the position index of the local window lt centered on position (h, w) and NNin is the local window size. Given that artifacts typically show up with high-frreuence characteristics, areas with higher variances likely indicate where these artifacts are. We could apply the artifact map weight to the image reconstruction loss as Lree = M @ (x -1112.
Our total loss to train our model in pixel space is follows:
Cpix = Lree + aLdis + 8.per
(4)
where a = 1.0 and 3 = 0.5 are weighting parameters, the discriminator loss Lais and the perceptual loss Cper are adopted to improve the overall quality of reconstructed images.
3.3 Semantic-aligned latent representation
In Stable Diffusion [34]. descriptions are incorporated into the latent representation using Cross- Attention (CA) blocks to ensure flexibility. However, this approach leads to the ghosting effect as color features are assigned outside the intended regions. To address this, we focus on maintaining semantic alignment between descriptions and grayscale images within the latent space.
In Fig. 2 (b), we replace Vanilla Convolution (VC) blocks in the noise prediction network with our Channcl-Extended Convolution (CEC) blocks within each downsampling module. These CEC blocks receive the resized luminance features i from the luminance encoder in the pixel space as additional guidance. By leveraging extended channels, CEC blocks could effectively capture the local structural semantics of the luminance in the latent space. We formulate the channel-extenue convolution as:
Nv-1 Nx-1
Nax
Next Σ
fh
Σ
fix
S
(5)
where ht w.P 1 N2/2 and + N1/2 2 are position index of feature maps. and I' are scparately feature maps before and after performing the convolution, Ng is the kernel size, Nix and Next separately mean the number of fixed channels and extended channels, glum denote resized luminance features. Mathematically, the CEC block is equivalent to using a stack of convolutions to extract features and add their output to the downsampling block. To preserve the robust language understanding and rich color priors of the pretrained generative model, we keep the weights fix fixed during training. Additionally, initialize the weights of extended channels ext to zero, ensuring that our model maintains functional equivalence to the pretrained generative model prior to training.
We use skip connections between downsampling and upsampling modules to guide the upsampling process with luminance features. To expedite training convergence, we utilize Convolution (CEC) blocks only in downsampling modules.
5

---

Table I: Quantitative comparisons with language-basse based colorization methods (left) and automatic colorization methods (right). marks our method receiving scarce-level descriptions. Throughout this paper, 1 (J) means higher (lower) is better. Best performances are highlighted in bold.
Compurison with language- bused colorization methods
Comparison with automatic colorization methods
Extended COCO-Stuff
Multi-instance
Extended COCO-Stuff
Multi-instance
Method
Method
PSNR1 SSIM LPIPS.
PSNR SSIM LPIPS!
PSNR1
SSIM LPIPS]
PSNR
SSIM1
LPIPS.
LBIE
22.02
.8519
.265
21.92
.8697
.260
CIC
22.03
.8893
21.95
.8754
.239
ML2018
21.06
8533
.282
20.54
8495
.294
InstColor
23.76
8979
.195
23.59
8968
.199
Xie2018
21.92
.8664
.233
20.04
.5300
.268
ChromaGAN
22.08
.3166
.275
22.41
.8884
.248
L-CoDe
24.86
.9911
.166
23.84
9056
.176
BigColor
21.45
8868
.236
21.43
8824
.225
L-CoDer
25.03
.9225
.162
24.16
9135
.168
DISCO
20.62
.844
.212
20.81
.8722
.204
L-Colns
25.10
.9511
.159
24.74
.9111
.156
CT
23.16
.9030
.191
22.65
.8993
.193
W/o LIC
23.87
.8866
.190
23.96
.9047
.160
LIC
22.25
.8584
22.53
.8697
.176
Wio SLR
17.26
.5944
.587
15.61
.4879
.531
W/o SLR"
15.73
.6132
.512
14.18
.5732
.472
W/o ISS
25.32
.9174
147
24.57
.9138
.149
W/o ISS*
24.24
.9073
.159
24.07
.9054
.165
L-CAD
25.97
.9191
.142
25.51
.9156
.127
L-CAD*
24.33
.9114
.157
24.19
.9088
.164
The loss for training the denoising network in the latent space is:
Liat e= [lke co(2t, t, yeex, , Jum
)
(6)
where ytex is any-level descriptions encoded by the text encoder of CL.IP (31
3.4 Instance-aware sampling strategy
To ensure accurate color assignment to objects in images with diverse and complex descriptions, propose an instance-aware sampling strategy. This strategy draws inspiration from the effectiveness of InstColor (39], which employs an auxiliary recognition model to enhance instance- performance. By incorporating the instance-aware sampling strategy into our colorization process, we aim to improve the accuracy and effectiveness of assigning colors to corresponding objects.
To improve mastance-wwareness, we employ referring segmentation model (e.g.. SAM 21 to estimate object contours mentioned in the description (Mest), Attention maps (MA")) at the 1-th CA block control colorized regions for each color word. We normalize the attention maps with Sigmoid and refine them iteratively by aligning with downsampled estimated contours. After that, apply the denoising process in DDIM 388. Algorithm ! outlines our instance-aware sampling strategy.
Algorithm 1: Instance-aware sampling strategy input Roughly estimated object contour Mest output: : :oloriized latent representation 20
Our instance- aware sampling strategy does heavily rely on precise seg- mentation results from the referring segmentation model. This is due to the following reasons: (i) Our sam- pling is performed in the latent space using low-resolution segmentation re- sults, mitigating the impact segmen- tation imprecision.
for t = T'............ do
.Matt ca co t, Jum (0x)
for = ... L do
Mest +- Downsampling( (Mest .()
M < Sigmoid(
(ii) When decod- ing the latent representation into the pixel space, the compression decoder, guided by local structural semantics from grayscale images, could adap- tively fix color bleeding issues to pro- duce visually pleasing results.
Maas +- Ma XV L.C Mest
end
é = co(2,, t. y y (Matr + Maat
2: DDIM(E,,
end
3.5 Learning and implementation details
During training, we utilize the classifier-free guidance [4 to supervise the colorization process conditioned on descriptions. To increase robustness for any-level descriptions, randomly replace 30% of the complete-level and partial-level descriptions with scarce-level descriptions. The pixel space modules are trained using Eq. with batch size of 8 for 10 epochs until convergence.

---

the groen, owo cheirs on the rightt are Lyenu, and they are on brown carpet.
Completo lovel
wearing a blacks and red le, and the man on the right is weering red suit and Black tie.
orange cetis sitting on a grean Slanket.
Pontica level
A double decker red uniii and purple bus.
Description
Grayscale
Ground Truth
ML2018
L-CoDe
L-Colns
O urs
Figure 3: Compared with language-base based colorization methods, our model achieves correct instance- aware colorization while improving the colorized quality for unmentioned objects.
Scafte leved
Grayscale
Ground Truth
CIC
BigColor
DISCO
CT?
Ours
Compared with automatic colorization methods, model shows comparable performance with searce-level descriptions that lack meaningful color information.
Figure 4:
Subsequently, we fix their weights and train the noise prediction network in the latent space using Eq. (6) for 50 epochs. The training process takes approximately 60 hours and is performed on NVIDIA GeForce 3090Ti graphic cards. We employ the Adam optimizer with learning rate of 1 x 10 for the pixel space and 5 10-5 for the latent space to minimize the losses. For sampling, we use the DDIM sampling method 38 with sampling steps.
4 Experiment
We conduct our experiments on language-basee based colorization datasets: (i) the extended COCO-Stuff dataset [45], which is built upon the COCO-Stuff dataset (3] by discarding unqualified samples for the colorization task, including 59K training images and 2.4K evaluation images; and (ii) multi-instance dataset (6) which provides samples featuring multiple instances with different visual characteristics within a single image, including 65K training images and 7K evaluation images. For both datasets, each image accompanied by a corresponding language description.
Dataset.
4.1 Comparison with state-of-the-art methods
We make comparisons with language-based colorization methods (e.g., LBIE (7). MI.2018 27) Xie2018 49), L-CoDe [45), L-CoDer [5], and L-CoIns (6) and automatic colorization methods

---

Table 2: User study results. marks our method receiving scarce-level descriptions. Ours (L-CAD) clearly produces a higher score than state-of-the-art methods on both datasets.
Corresponding experiment
Reality experiment
Method
Extended COCO-Stuff
Multi-instance
Method
Extended COCO-Stuff
Multi-instance instance
LBIE
1.68%
3.20%
CIC
6.24%
3.76%
ML2018
3.28%
4.92%
InstColor
11.68%
10.36%
Xic2018
0.96%
1.72%
ChromaGAN
7.16%
6.32%
L-CoDe
11.56%
11.32%
BigColor
11.48%
9.56%
L-CoDer
16.32%
15.24%
DISCO
8.92%
10.84%
L-CCols
24.76%
19.56%
CT2
13.36%
12.92%
Ours
41.44%
44.04%
Ours
19.72%
20.28%
Ground truth
N/A
N/A
Ground truth
21.44%
25.96%
(e.g., CIC [52]] InstColor [39] ChromaGAN 40 BigColor [20], DISCO 47 and CT demonstrate that our model capable of effectively handling any-level descriptions.
44 to
Qualitative comparisons. Previous language-baseke colorization methods assume that users provide comprehensive color descriptions, resulting in suboptimal performance, particularly for objects without corresponding color descriptions. In contrast, leveraging the prior knowledge of Stable Diffusion
M and our newly proposed instance-aware sampling strategy, our L-CAD presents vivid colorization results with any-level descriptions, as shown in Fig. we further make comparisons with automatic colorization methods using scarce-level descriptions that lack meaningful color information. As shown in Fig. 4 our model still produces superior colorization results.
we Peak Signal-to-Noise Ratio 42 and Learned Perceptual Image Patch Similarity (LPIPS) (53] qualitatively evaluate the quality of colorization. With complete-level and partial-level descriptions, our method (L-CAD) outperforms all language-based colorization methods. Furthermore, with scarce-level descriptions, our method shows distinct advantage over automatic colorization methods. As shown in Tab. I our method achieves the best PSNR, SSIM, and LPIPS scores on both datasets.
Quantitative comparisons. Following previous works 6 (PSNR) [6). Structural Similarity Index Measure (SSIM)
45
4.2
User study
We further conduct user studies to evaluate the subjective perception of human observers: (i) Cor- responding experiment: We apply language-based colorization methods to colorize images with random complete-level or partial-level descriptions. Participants are instructed to select the image that best matches the description. (ii) Reality experiment: Participants are asked to select the image they perceive to be the most visually realistic among the real image, images colorized by our method with scarce-level descriptions, and those colorized by automatic colorization methods. For each experiment, we randomly select 100 samples from the testing images of each dataset, and conduct them independently by 25 volunteers the Amazon Mechanical Turk (AMT). As shown in Tab. our method achieves the highest scores both experiments.
4.3 Ablation study and discussion
We create three baselines to study the impact of our proposed modules and the sampling strategy. The evaluation scores and colorization results of the ablation study are shown in Tab. and Fig. [
W/o Luminance-guided Image Compression (LIC). We disable the luminance encoder in the pixel space and revert to using the perceptual compression model in (34) Without the guidance of multi-scale grayscale features, the model cannot correctly preserve local spatial structures.
W/ Semantic-alignet Latent Representation (SLR). We remove luminance features that guide the semantic alignment in the latent space and therefore replace channel- convolution blocks with vanilla convolution blocks. As result, the obvious ghosting effect occurs in colorized images.

---

Complete lnvel
The woman on the left wears plnnk, woman on the dark
Parttal level
Yellow shirted tennis player looking for incominn ball.
1.12
1.12
1.12
1.12 ILEX
1.12
Scarce lovel
A colorful Image.
Description
Grayscale
Ground Truth
W/o LIC
W/o SLR
Wo IS
Ours
Figure 5: Ablation study. When proposed modules are disable, results become counterintuitive.
Complete level
There is yellow green ball the dark blue tennis racket.
G
Parthal leval
A blue sign light top.
Description
Grayscale
Ground Truth
ControlNet
Pix2PixZen ixzero
SDE
Ours
Figure 6: Compared with text-driven editing methods, our model colorizes images to be spatially aligned with grayscale images.
W/o Instance-aware Sampling Strategy (ISS). We replace the instance-aware sampling strategy with the standard DDIM (38), which significantly degrades the performance of the model correctly assign colors to corresponding objects that have different descriptions.
Discussion. Recent studies (e.g., ControlNet [50). Pix2PixZero [30], and SDEdit 28 could also
edit images with descriptions, based on the pretrained generative models. However, these methods not tailored to the colorization task, resulting in challenges related to preserving local spatial structures, leveraging rich color priors, and learning correspondence between objects and color words. These limitations make it difficult for them to perform instance-aware colorization with any-level descriptions. We present their unsuitability in Fig.
4.4 Evaluation on ImageNet dataset
We conduct extensive experiments on the widely-used ImageNet dataset [36], including 1.3M images covering 1000 categories, to evaluate the performance of our L-CAD for automatic colorization. To provide comprehensive evaluation, introduce the Fréchet inception distance [12] to quantify the distribution gap between colorized images and ground truth images, as well as the colorfulness score
||] to reflect the vividness of colorized images. Following previous works 2 22 44, we evaluate
our model on the first 5K images of the public validation set, where all the test images are center cropped and resized into 256 256 resolution.
As shown Tab. 3 our model achieves top scores in FID, PSNR, SSIM, and LPIPS metries, and demonstrates comparable performance in colorfulness metrics. These results indicate that model aims to achieve photo-realistic colorization, emphasizing natural tones rather than exaggerated saturation. It should be noted that due to the absence of provided descriptions in the ImageNet

---

Table 3: Quantitative comparison with more automatic colorization methods on ImageNet dataset. marks our method receiving scarce-level descriptions.
Method
FID
PSNR
SSIM
LPIPS_
colorfult
Acolorful
CIC [52]
8.72
22.64
0.91
0.22
31.60
4.72
DeOldify 1
9.45
21.12
0.83
0.24
22.70
13.62
ChromaGAN
40
7.66
23.35
0.90
0.21
27.88
8,43
InstColor
39
8.06
23.28
0.91
0.21
24.87
11,44
GCP
5.95
21.68
0.88
0.23
32,98
3.34
BigColor
21.35
0.87
0.25
30.60
DSICO
7.99
20.31
0.86
0.23
48.43
10.70
ColTran
6.44
20.95
0.80
0.29
34.50
2.24
CT 4+
5.51
23.50
0.92
0.19
38.48
2.17
ColorFormer
18
4.64
23.14
0.89
0.18
37,95
0.23
I-CAD*
4.36
24.47
0.92
0.16
34.04
3.68
"Japemeso
There' black
1906. Creek
curtains behind them golddnn.
resideniaal block Main Cabee"
A colortul
Calffrria."
the
Mashingtan
Figure 7: Examples of applying our method to colorize legacy photos with any-level descriptions.
dataset (3), model is trained solely with scarce-level descriptions (e.g., "a colorful image") and leverage the rich image semantics encapsulated in language- descriptions by initializing weights pretrained the extended COCO-Stuff dataset (45). This deviates from the training strategy
adopted in language-basse based colorization datasets 6 45 (detailed in Sec. 3.5). potentially affecting
the performance of our model.
4.5 Application
We apply our method to colorize legacy Black-and-whin photos using descriptions at the complete. partial, and scarce levels, respectively. Results presented sequentially from left to right in Fig. which demonstrates the robust generalization capability of our model.
Conclusion
We present L-CAD, unified model to perform Language-based Colorization with Any-level Descriptions. Leveraging the prior knowledge of the pretrained model, we design novel modules preserve local spatial structures and prevent the ghosting effect. we further propose the instance- aware sampling strategy to colorize diverse and complex scenarios. We demonstrate our model could handle any-level descriptions by making comparisons with language-basee colorization methods with complete-lewe and partial-leve descriptions, as well as automatic colorization methods with scare- level descriptions. Both qualitative and quantitative results demonstrate our superior performance.
Limitations. Our model as a diffusion model takes an additional sampling process for visualizing colorization results. This requirement considerably slows down the generation speed, and further limits the potential for real-time applications our model. To address this concern, adopting advanced fast sampling methods (e.g., DPM-Solver
26 could be promising direction.
Acknowledgement. This work supported by the National Key R&D Program of China under Grant No. 102020000,000, the National Natural Science Foundation of China under Grand No. 62136001, 62088 102, and Program for Youth Innovative Research Team of BUPT No. 2023QNTD02. Yu Li is supported by Shenzhen Hetao Seaaazzhen-oonge Kong Science and Technology Innovation Cooperation Zone, under Grant No. HTHZOSWS-KCCOO XCCYB-202B32222 2023052.
10

---

References
J. Antic. A deep learning based project for colorizing and restoring old images (and video!).
[2] L. Ardizzone, C. Lüth, J. Kruse, C. Rother, and U. Köthe. Guided image generation with conditional invertible neural networks, arXiv preprint arXiv:1907.02392. 2019.
[3] H. Cacsar, Uijlings, and V. Femari. COCO-Stuff: Thing and stuff classes in context. In CVPR, 2018.
[4] Y. Cao. Z. Zhou, W. Zhang, and Y. Yu. Unsupervised diverse colorization via generative adversarial networks. In ECMI-PKKD), 2017.
[5] Z. Chang, S. Weng, Y. Li, S. Li, and B. Shi. L-CoDer: Language-basse colorization with color-object decoupling transformer. In ECCV, 2022.
[6] Z. Chang, S. Weng. P. Zhang, Y. Li, S. Li, and B. Shi. L-Colns: Language-bass based colorization with instance awareness. In CVPR, 2023.
[7] J. Chen, Y. Shen, J. Gao, J. Liu, and X. Liu. Language-based image editing with recurrent attentive models. In CVPR, 2018.
[8] Z. Cheng, Q. Yang, and B. Sheng. Deep colorization. In ICCV, 2015.
[9] A. Deshpande, J. Lu, M.-C. Yeh, M. Jin Chong, and D. Forsyth. Learning diverse image colorization. In CVPR, 2017.
[10] S. Guadarrama, R. Dahl, D. Bieber, M. Norouzi, J. Shlens, and K. Murphy. PixColor: Pixel recursive colorization, arXiv preprint arXiv:1705.07208, 2017.
[11] D. Hasler and S. E. Suesstrunk. Measuring colorfulness in natural images. In Human vision and electronic imaging VIII, 2003.
[12] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter. GANs trained by time-scale update rule converge nash equilibrium. In NIPS, 2017.
[13] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion prohabilistic models. In NIPS, 2020.
[14] J. Ho and T. Salimans. Classifier-free diffusion guidance. In NeuriPS 2021 Workshop on Deep Generative Models and Downstream Applications.
[15] Z. Huang. N. Zhao, and J. Liao. UniColor: A unified framework for multi-modal colorization with transformer. In SIGGRAPH Asia, 2022.
[16] Q. Huynh-Thon Thu and M. Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics letters, 2008.
[17] S. lizuka, E. Simo-Serra, and H. Ishikawa. Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ToG, 2016.
[18] X. Ji, B. Jiang, D. Luo, G. Tao, W. Chu, 7. Xie, C. Wang, and Y. Tai. ColorFormer: Image colorization via color memory assisted bybrid-attention transformer. In ECCV, 2022.
[19] B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
[20] G. Kim, K. Kang, S. Kim, H. Lee, S. Kim, J. Kim, S.-H. Back, and S. Cho. BigColor: Colorization using generative color prior for natural images. ECCV, 2022.
[21] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.0266 2023.
[22] M. Kumar. D. Weissenborn, and N. Kalchbrenner. Colorization transformer. In ICLR, 2021.
[23] G. Larsson, M. Maire, and G. Shakhnarovich. Learning representations for automatic colorization. In ECCV, 2016.
[24] B. Li, X. Qi, T. Lukasiewicz, and P. H. Tor. ManiGAN: Text-guided image manipulation. In CVPR, 2020.
[25] J. Liang. H. Zeng. and L. Zhang. Details or artifacts: A locally discriminative learning approach to realistic image super-resolution. In CVPR, 2022.

---

[26] C. Lu, Y. Zhou. F. Bao. J. Chen, C. Li, and J. Zhu. DoM-Solver: A fast ODE solver for diffusion probabilistic model sampling around 10 steps. In NIPS, 2022.
[27] V. Manjunatha, M. lyyer, J. Boyd-Graben and L. Davis. Learning color from language. In NAACL, 2018.
[28] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2021.
[29] T. Miyato and Koyama. CGANs with projection discriminator. In ICLR, 2018.
[30] G. Parmar, K. K. Singh, R. Zhang, Y. Li, J. Lu, and J.-Y. Zhu. Zero-shot image-to-image translation. arXiv preprint arXiv:2302.00000 030077, 2023.
[31] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[32] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[33] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In ICMI., 2021.
[34] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[35] N. Ruiz, Y. Li. V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
[36] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, al. Imagenet large scale visual recognition challenge. I/CV, 2015.
[37] C. Saharia, W. Chan, Saxena. L. Li, J. Whang. E. L. Denton, K. Ghasemipour, R. Gontijo Lopes. B. Karagol Ayan, Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NIPS, 2022.
[38] Song, C. Meng, and Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[39] J.-W. Su, H.-K. Chu, and J.-B. Huang. Instance-aware image colorization. In CVPR, 2020.
[40] P. Vitoria, L. Raad, and C. Ballester. ChromaGAN: Adversarial picture colorization with semantic class distribution. In WACV, 2020.
[41] Y. Wang, M. Xia, L. Qi. Shao, and Y. Qiao. PalGAN: Image colorization with palette generative adversarial networks. In ECCV, 2022.
[42] Z. Wang, C. Bovik, H. R. Sheikh, and E, P. Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004.
[43] S. Weng, W. Li, D. Li, H. Jin, and B. Shi. Conditional image repainting via semantic bridge and piecewise value function. In ECCV, 2020.
[44] S. Weng, J. Sun, Y. Li, S. Li, and B. Shi. CT Colorization transformer via color tokens. In ECCV, 2022.
[45] S. Weng, H. Wu, Z. C. Chang, J. Tang. S. Li, and B. Shi. L-CoDe: Language-basse colorization using color-object decoupled conditions. In AAAI, 2022.
[46] Y. Wu, X. Wang, Li, H. Zhang, X. Zhao, and Y. Shan. Towards vivid and diverse image colorization with generative color prior. In ICCV, 2021.
[47] M. Xia, W. Hu, T.-T. Wong, and J. Wang. Disentangled image colorization via global anchors. TOG, 2022.
[48] S. Xie, Z. Zhang, Z. Lin, T. Hinz, and K. Zhang. SmartBrush: Text and shape guided object inpainting with diffusion model. In CVPR, 2023.
[49] Y. Xie. Language-guided image colorization. Master's thesis, ETH Zurich, Departement of Computer Science, 2018.
[50] L. Zhang and M. Agrawala. Adding conditional control text-to-image diffusion models, 2023.
12

---

[51] L. Zhang. Q. Chen, B. Hu, and S. Jiang. Text-guided neural image inpainting. In ACM MM, 2020.
[52] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In ECCV, 2016.
[53] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unrcasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[54] Y. Zhang, N. Huang. Tang, H. Huang, C. Ma, W. Dong, and C. Inversion-hascd creativity transfer with diffusion models. In CVPR, 2023.
[55] Zhao, Han, L. Shao, and C. G. Snock, Pixelated semantic colorization. IJC,, 2020.
[56] Zhao, Liu, G. Snoek, Han, and Shao. Pixel-Ievel semantics guided image colorization. In BMVC, 2018.
13


Reviews:
[{"review_id": "pLoEAdSWhJ", "reviewer": "Reviewer_Gu3m", "summary": "The paper addresses the problem of colorizing images with descriptions of diverse levels of details. The key idea of the work is to propose a unified model that adaptively understands any-level descriptions by leveraging the pretrained cross-modality generative model. Additionally, the paper introduces modules that aid in preserving local spatial structures and prevent the ghosting effect by aligning with input conditions in both the pixel space and the latent space. Further, the paper presents an instance-aware sampling strategy to correctly assign colors to corresponding objects, enabling effective colorization in diverse and complex scenarios. The work demonstrates state-of-the-art performance in both automatic and language-based colorization methods. ", "questions": "Please see weaknesses.", "limitations": "Authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and well-motivated. Qualitative and quantitative comparisons to existing methods show the effectiveness of the proposed approach in language based colorization and automatic colorization. User studies and ablation studies sufficiently justify the key conclusions made in the paper.\n", "weaknesses": "One of the key weaknesses of the paper is the readability of a few sub-sections. For example, I struggle to understand the detailed implementation/intuition for section 3.2 where the paper introduces luminance guided image compression and how it helps preserve local structural semantics. It is not very intuitive as to how it can help bridge alignment between colorization results and grayscale images. Some more detailed intuitions could help readers to better follow this section. Similarly the justifications for equation 3 could be further elaborated. In lines 153-154, on the ghosting effects, I am curious to know if this is still the case with using region based guidance (for example, see below two references). I will revise my scores based on the clarifications in the rebuttal.\n\nYang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., ... & Wang, L. (2023). Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14246-14255).\n\nLi, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., ... & Lee, Y. J. (2023). Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22511-22521).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their insightful and constructive feedback.\n\nAs the rebuttal instruction, Fig. S1-S13 and Tab. S1-S2 are shown in the PDF attached to the global response.\n\n### ***Weaknesses***\n- Q: Detailed implementation/intuition for Sec. 3.2.\n\nOur primary intuition is to leverage Stable Diffusion [31]’s robust language understanding and rich color priors for language-based colorization with any-level descriptions (L125-126).  To realize this goal, we analyze the architecture of Stable Diffusion. This method adopts a compression encoder $\\mathcal{E}$ to encode image $x$ from pixel space into latent space as $z = \\mathcal{E}(x)$, and a compression decoder $\\mathcal{D}$ to reconstruct the image as $\\tilde{x}=\\mathcal{D}(z)$ (L118-122). A critical observation is that Stable Diffusion lacks the ability to preserve local spatial structures of input grayscale images (L128-129). To remedy this limitation, we propose an additional luminance encoder $\\hat{\\mathcal{E}}$ in the pixel space as a bridge to align colorization results with grayscale images (L130-132). As shown in Fig. 2 (a), luminance encoder extracts multi-scale features from grayscale images, which preserves local structural semantics of grayscale images $\\hat{\\mathcal{E}}(x^{\\mathrm{lum}})$ (L133-135). These features are added directly into the compression decoder, guiding its decoding process (L135-136).\n\nWe implement the architecture of luminance encoder $\\hat{\\mathcal{E}}$ mirrors that of the compression encoder $\\mathcal{E}$ of Stable Diffusion. The weights of compression encoder $\\mathcal{E}$ and compression decoder $\\mathcal{D}$ are fixed to retain prior knowledge from the pretrained model (L136-138). A visualization of the architecture for both the compression encoder and decoder is provided in Fig. S1.\n\nWe will revise this in the final version for better readability.\n\nWe will release the code, offering a more comprehensive understanding once the paper is accepted.\n\n- Q: Equation 3 could be further elaborated.\n\nWhen training the luminance encoder $\\hat{\\mathcal{E}}$, our goal is to identify and minimize the discrepancy between colorization results and grayscale images (L140-141). Since we observe that erroneous pixels significantly damage visual perception (L139-140), we intend to estimate an artifact map $M^\\mathrm{art}_{h,w}$. This map would serve to indicate the probability of encountering artifacts at specific spatial position $(h,w)$ within the colorized results $\\tilde{x}$. Specifically, we calculate the residual between the ground truth image and the colorized result as $\\delta = x - \\tilde{x}$ (L142-143). Next, we compute the variance of the aforementioned residual within local square windows at each position (L143-144),  as shown in Eq. 3.  Given that artifacts typically show up with high-frequency characteristics, areas with higher variances likely indicate where these artifacts are. Finally, we apply the artifact map as a weight to the image reconstruction loss (L146-147) to focus on the pixels where the model needs to further minimize the discrepancy.\n\nWe further visualize the artifact map with different $N_{\\mathrm{win}}$ to demonstrate the effectiveness of estimated artifacts map, as shown in Fig. S2.\n\nWe will revise this in the final version for better readability.\n\n- Q: Using region-based guidance.\n\nTo investigate whether object boxes could effectively mitigate the ghosting effects, we conduct two additional ablation studies by replacing modules in the latent space with the corresponding components from reference [R1] and [R2]. These modifications are individually designated as \"L-ReCo\" and \"L-GLIGEN\". To evaluate these ablation studies on the extended COCO-Stuff and Multi-instance datasets, we employ DINOv2 [R3] to estimate object boxes. While the results indicate a reduction in ghosting effects, they remain visible. This is because object boxes could only offer a coarse-grained location of primary objects, which is far less precise compared to the fine-grained luminance features provided by grayscale images. We show qualitative and quantitative results in Fig. S3 and Tab. S1, respectively.\n\n[R1] Z Yang, J Wang, Z Gan, L Li, K Lin, C Wu, N Duan, Z Liu, C Liu, M Zeng, and L Wang. ReCo: Region-controlled text-to-image generation. In *CVPR*, 2023 .\n\n[R2] Y Li, H Liu, Q Wu, F Mu, J Yang, J Gao, C Li, and YJ Lee. GLIGEN: Open-set grounded text-to-image generation. In *CVPR*, 2023.\n\n[R3] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby,\net al. DINOv2: Learning robust visual features without supervision. *arXiv preprint arXiv:2304.07193*, 2023."}, {"review_id": "ihwiQWPokZ", "reviewer": "Reviewer_Gyr2", "summary": "This paper presents a well engineered method for Stable\nDiffusion-based text-guided image colorization. The paper addresses\nthe challenge of structural fidelity to the original grayscale image\nby additional conditioning modules in the denoiser architecture. The\npaper also proposes an instance-aware sampling procedure that uses\nobject segmentations to encourage accurate color assignment. Extensive\nquantitative and qualitative comparison show the proposed method\noutperforms the compared baselines.\n", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is a well engineered method. The design decisions appear\n  reasonable and are justified with ablation studies.\n- The paper is well written, the ideas are clear and easy to understand.\n- The qualitative results are remarkable and quantitative comparisons\n  show the advantage of the proposed method in standard benchmarks and\n  user preference studies.\n", "weaknesses": "- The colorization baselines in Figure 1 and 4 look much weaker than\n  in the original papers (e.g. [43] Fig. 6 and [40] Fig. 5). I wonder\n  how the proposed method would compare in the referenced figures.\n- It would be interesting to include in the quantitative comparison\n  the number of sampling steps required for each method, as this is a\n  disadvantage of the diffusion-based methods compared to the GAN-based\n  ones.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for this thoughtful review and we are glad to see their positive assessment. \n\nAs the rebuttal instruction, Fig. S1-S13 and Tab. S1-S2 are shown in the PDF attached to the global response.\n\n### ***Weaknesses***\n- Q: Colorization baselines look much weaker.\n\nFor the performance assessment of [40] and [43], we adopt the publicly available pretrained weights (*i.e.*, [https://github.com/MenghanXia/DisentangledColorization](https://github.com/MenghanXia/DisentangledColorization) and [https://github.com/shuchenweng/CT2](https://github.com/shuchenweng/CT2)), which ensures a transparent and objective comparison.\n\nThe scenarios displayed in Fig. 1 and Fig. 4 are particularly challenging due to the abundance of small objects with intricate textures.  In our observations, when objects in grayscale images present recognition difficulties, [40] and [43] tend to produce undersaturated colorization results. Leveraging the robust language understanding and rich color priors of Stable Diffusion [31], our model could adeptly achieve vivid colorization, even in such challenging cases. To further show the superior performance of our proposed method, we present supplementary qualitative experiments against [40] and [43] in typical outdoor scenarios where they have been known to perform satisfactorily. Please refer to Fig. S4.\n\n- Q: The number of sampling steps.\n\nIt is noteworthy to point out that among the methods enumerated in Tab. 1 and Tab. 3, ours is the only one based on diffusion models. \n\nTherefore, we conduct an additional ablation study focusing on the impact of varying the number of sampling steps $N_\\mathrm{step}$.  The corresponding evaluation scores and colorization results are presented in Tab. S1 and Fig. S5, respectively. \n\nFurthermore, we demonstrate that our method could be accelerated by adopting advanced fast sampling methods (*i.e.*, DPM-Solver++ [R1]) while maintaining high-quality colorization results, as shown in Fig. S5.\n\n[R1] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. DPM-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2023."}, {"review_id": "MnOosaQ8MV", "reviewer": "Reviewer_whze", "summary": "This paper presents a novel approach to image colorization, which demonstrates superior performance in language-based colorization methods. The proposed model employs language descriptions at varying detail levels to produce high-quality, customizable colorized images by diffusion in the latent space. A key innovation is L-CAD's adaptive understanding of any-level descriptions, facilitating precise colorization based on user requests. To ensure proper spatial alignment with grayscale inputs and avoid ghosting effects, a luminance-guided compression module, and a channel-extended convolution operator are introduced. An instance-aware sampling strategy was adopted from previous literature to enhance color assignment to objects. L-CAD exhibits very good results in both quantitative and qualitatve experiments. ", "questions": "- Can the authors please compare training and inference time between their method and the baselines?", "limitations": "- The authors have mentioned that their model is relatively slow.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. \n- The problem that they tackle is interesting and valuable. \n- The results are promising and the ablation studies to disentangle different modules have been conducted.", "weaknesses": "- The method is a little complex and needs multiple modules to work properly. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and suggestions.\n\nAs the rebuttal instruction, Fig. S1-S13 and Tab. S1-S2 are shown in the PDF attached to the global response.\n\n### ***Weaknesses***\n- Q: The method is a little complex.\n\nPrevious works [4,5,6,24,41] implicitly assume that users provide comprehensive color descriptions for most of the objects in the image, which often leads to suboptimal performance, especially for objects without corresponding color descriptions (L24-26). To address the aforementioned issue, we intend to utilize Stable Diffusion [31]'s robust language understanding for mentioned objects and rich color priors for unmentioned ones (L38-41). Therefore, the model needs to be elaborately designed to be aligned with Stable Diffusion in the pixel space, the latent space, and the sampling strategy.\n\nAs illustrated in Sec. 4.3, we demonstrate that every component of our model is indispensable for effectively performing language-based colorization with any-level descriptions.\n\n### ***Questions***\n- Q: Training and inference time.\n\nThanks for the suggestions, we present the training and inference time of all methods in Tab. S2. \n\n### ***Limitations***\n- Q: Model is relatively slow.\n\nAs illustrated in L278-279, this limitation could be mitigated by using advanced fast sampling methods (*i.e.*, DPM-Solver++ [R1]).  We show qualitative and quantitative results using DPM-Solver++ in Fig. S5 and Tab. S1 of the attached PDF, respectively. The accelerated inference time is presented in Tab. S2. \n\n[R1] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. DPM-Solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2023.\n\n"}, {"review_id": "rLv6PlROr2", "reviewer": "Reviewer_ZuAx", "summary": "The paper proposes a modification over Stable Diffusion to adapt it to perform language-based colorization of images with varying levels of description details.\nThe method is based on three modifications. First, in addition to the auto-encoder used for SD, the authors add another encoder that is tasked with preserving the spatial features of the input image. Second, the convolutions in the downsampling layers of the UNet are replaced with a novel Channel-Extended Convolution to encourage the reliance on the encoded spatial features. Finally, a segmentation model is employed to encourage correct object-level color assignment.\nExtensive experiments are conducted to demonstrate the method's superiority over both language-conditioned colorization methods and automatic colorization methods.", "questions": "- The authors mention the \"ghosting effect\" several times, but no explanation is given as to what it is (e.g., L. 153).\n- Why is the CEC block only used in the first half of the UNet (downsampling)?\n- How robust is the method to OOD outputs? For example, what happens if you train on COCO-Stuff and evaluate on multi-instance?", "limitations": "Limitations are discussed briefly. The reviewer thinks that a more in-depth comparison of runtime to SD and to the baselines is needed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors conduct very extensive experiments, using various datasets and human evaluation.\n- Overall, the reviewer found the qualitative results to be convincing.\n- The idea of slightly modifying the Stable Diffusion architecture for other tasks is interesting and can possibly be generalized to other tasks.", "weaknesses": "Please note that the low confidence is due to the fact that the reviewer is not familiar with literature on image colorization, and therefore feels less confident in giving an assessment of this work. \n\nReadability:\nOverall, the reviewer found the method section to be a bit hard to follow. Intuitions are lacking in some parts, and familiarity with previous works is required to follow the explanations.\n- The writing of the preliminaries section on diffusion models is a bit confusing and inaccurate. For example, it starts with describing the inference (L. 106-107) but then turns to describe the training process (Eq. 1 and on).  \n- Section 3.3 (Channel-Extended Convolution) is not entirely clear. A supporting figure demonstrating the VC vs. CEC would be beneficial to better understand the module, its motivation and novelty.\n- Section 3.4 heavily relies on familiarity with existing colorization methods, and the manipulation to the cross-attention maps is not clear to the reviewer (e.g., why employ a sigmoid non-linearity on the attention?) \n\nEvaluation:\n- The authors claim that when ablating the Instance-aware Sampling Strategy (ISS) \"significantly degrades the performance of the model to correctly assign colors to corresponding objects that have different descriptions.\" (L.256) however, the results in Fig. 5 and Tab. 1 appear to be pretty similar to the proposed method.\n- The comparison to image editing methods are partial, however given the extensive comparisons, this is not a major concern.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and valuable feedback.  \n\nGiven the character limit (6000), we have to make our response brief. For additional details, we welcome a more comprehensive discussion during the Author-Reviewer Discussions.\n\nAs the rebuttal instruction, Fig. S1-S13 and Tab. S1-S2 are shown in the PDF attached to the global response.\n\n### ***Readability*** \n- Q: Confused preliminaries.\n\nThanks for the suggestions. To enhance the clarity and comprehensibility of the preliminaries section, we will incorporate additional background information and strategically reorganize the arrangement of the paragraphs in the final version. \n\n- Q: Figure about VC vs. CEC.\n\nThe CEC block is designed to extend the channel number of vanilla convolution block so that the model could utilize the extended channels to effectively capture the local structural semantics of the luminance in the latent space (L159-160). By initializing the weights of extended channels to zero, CEC block ensures that our model maintains functional equivalence to the pretrained generative model prior to training (L166-168). \n\nWe provide a figure demonstrating the VC vs. CEC in Fig. S6 for better clarity.  In application, we feed the concatenation of resized luminance features and feature maps into our CEC block to extract joint feature maps. \n\n- Q: Relying on existing methods.\n\nPrevious works assume that users provide comprehensive color descriptions for most of the objects in the image, which causes suboptimal performance (L25-26). Given the inherent ambiguity in the number of objects mentioned in any-level descriptions (L38-39), we leverage the pretrained cross-modality generative model (*i.e.*, Stable Diffusion [31]) to utilize its robust language understanding for mentioned objects and rich color priors for unmentioned ones (L39-41). \n\n- Q: Cross-attention maps.\n\nWe apologize for typos in L193-194, which should be rewritten as:\n\nL193 $\\mathcal{M} \\leftarrow \\mathrm{Sigmoid}(M^\\mathrm{att}_l)$\n\nL194  $\\hat{M}^\\mathrm{att}_l \\leftarrow M^\\mathrm{att}_l - \\lambda \\nabla _\\mathcal{M} \\mathcal{L} _\\mathrm{BCE}(\\mathcal{M}, \\hat{M}^\\mathrm{est}_l)$\n\nWe briefly introduce the manipulation procedures applied to attention maps: *(i)* Matrix multiplication is firstly utilized to compute the unnormalized attention maps $M_l^\\mathrm{att} \\in \\mathbb{R}^{\\bar{h} \\times \\bar{w} \\times N_\\mathrm{obj}}$, where $\\bar{h}$ and $\\bar{w}$ are corresponding spatial resolution at $l$-th CA block, and $N_\\mathrm{obj}$ is the number of objects in the description. *(ii)* Then, we apply L193-194 to manipulate the attention maps $M^\\mathrm{att}_l$, utilizing sigmoid function to confine the value range of attention maps to $[0,1]$ — the same value range as that of the estimated object contours $M^\\mathrm{est}_l$. *(iii)* Finally, we apply a softmax operation to normalize the modified $\\hat{M}^\\mathrm{att}_l$. \n\nWe will revise this in the final version.\n\n### ***Evaluation*** \n- Q: Effectiveness of Instance-aware Sampling Strategy (ISS).\n\nThe performance improvement by adopting ISS is notable. \n\nAs presented in Tab. 1 (left), when provided with complete-level and partial-level descriptions, the absence of ISS (*W/o* ISS) results in a decline in PSNR from 25.97 to 25.32 (-0.65) on the extended COCO-stuff dataset. Furthermore, for the multi-instance dataset, which provides samples featuring multiple instances with different visual characteristics within a single image, the PSNR drop becomes larger from 25.51 to 24.57 (-0.94). Figure 5 reveals that *W/o* ISS may not correctly assign colors to corresponding objects that have different descriptions, *e.g.*,  the right woman in the first row being incorrectly colorized in pink instead of dark blue, and the man's white pants in the second row being colorized as yellow.\n\nIt is only in the context of scarce-level descriptions, which inherently lack meaningful color information for objects, that the model without ISS exhibits comparable performance. In these scenarios, the significance of ISS understandably diminishes.\n\n- Q: Comparison to image editing methods are partial.\n\nWe appreciate the reviewer's understanding. The primary objective of this study addresses persistent challenges in colorization task, *i.e.*,  colorizing images with descriptions of varying detail levels.  \n\n### ***Questions*** \n- Q: Ghosting effect.\n\nThis concept is recognized in the field of photography and image restoration [R1]. In the context of our work, the ''ghosting effect'' implies the model synthesizes an image resembling a composite created from multiple blended images. *E.g.*, in the second row of Fig. 5, *W/o* SLR appears to produce an image of a man in a yellow shirt and subsequently merge it with the original grayscale image. We will explain this term clearly in the final version.\n\n- Q: CEC block in downsampling.\n\nWe strategically apply CEC block in downsampling modules to expedite training convergence. Additionally, we conduct an ablation study where we replace VC blocks in both upsampling and downsampling modules with CEC blocks, denoted as ''upsampling''. The qualitative and quantitative results are shown in Fig. S7 and Tab. S1, respectively. Consequently, this ablation results in comparable performance.\n\n- Q: Robustness of OOD.\n\nWe train models on extended COCO-stuff dataset and subsequently evaluate it on the multi-instance dataset, and vice versa. We name this experiment as ''OOD'', and provide qualitative and quantitative results in Fig. S8 and Tab. S1, respectively. These results demonstrate a certain degree of robustness of our method.\n\n### ***Limitations*** \n- Q: In-depth comparison of runtime to baselines.\n\nWe include training and inference time for all methods in Tab. S2, which indicates our method cost more inference time. \n\nThis could be mitigated by using advanced fast sampling methods (DPM-Solver++).\n\n[R1] YC Shih, D Krishnan, F Durand, and WT Freeman. Reflection removal using ghosting cues. In *CVPR* 2015."}, {"review_id": "j3E2gCZL3z", "reviewer": "Reviewer_oFt4", "summary": "In this paper, the authors propose a model to adapt pretrained Stable Diffusion for language-conditioned colorization. In specific, they propose a luminance encoder that produces latents $y_lum$. This conditions the compression decoder and the denoiser of the diffusion model. The conditioning of the denoiser is done using a \"Channel Extended Convolution block\". Further during sampling, they utilize gradients from a semantic segmentation model to help with instance differentiation.\n\nComparisons are made to several language-conditioned and unconditional baselines across 2 datasets: extended coco-stuff and a (private) multi-instance dataset. They outperform these baselines on both quantitative metrics and mechanical turk. They show qualitative samples where they are able to colorize multi-level descriptions better than the baselines.", "questions": "**Most Important**: Amongst the standard training hyperparameters, the paper introduces $\\lambda$ for sampling, the loss hyperparameters $\\alpha$ and $\\beta$ and the number of elements $N_{win}^2$ in the window for the loss in Eq 3). How are these hyperparameters tuned? \n\n\nLuminance-guided image compression\n-----------------------------------------\n\n* The design of conditioning the “compression decoder” with the grayscale features makes sense. To be more self-conditioned, I suggest the authors to a) Briefly explain the architecture of the compression encoder + decoder b) Describe how exactly the conditioning from the luminance encoder features is incorporated in the compression decoder?\n* In Section 3.2, the authors introduce a discriminator loss but this is not explained or ablated. Is this introduced in this work are part of stable diffusion?\n* The authors introduce a loss that promotes smoothness by upweighting windows with high variance (Eq 3). The authors should ablate this weighting factor. Moreover, I think only the “luminance conditioning” is introduced in this work. So I suggest that the authors move the part that focuses on the training the compression encoder+decoder with the loss functions into the preliminaries and focus only on the luminance conditioning in this subsection.\n\nSemantic-aligned latent representation\n-------------------------\n\n* Imho, the novelty of this section is overclaimed. The “semantic-aligned” latent representation amounts to a zero-initialized convolution which takes as input to the grayscale features and is added to the output of denoising downsampling blocks. It is great that it works, but the authors should describe it as it is and not overclaim novelty.\n* Have the authors tried the obvious baseline that just concatenates $y_lum$ with $z_t$ at the inputs with downsampling to match the shapes?\n\nInstance-aware sampling strategy\n-----------------------\n\n* IIUC, the goal is to differentiate between different instances and if the output of a semantic segmentation model is used to provide this information as some sort of guidance or ground truth. If this is indeed the case, then the authors should ablate the simpler baseline, which is just to condition the denoising process with the output of a semantic segmentation model during training (as they do with the grayscale images)\n* What eactly is the semantic segmentation model used here? The authors say (eg SAM but it it not clear if SAM is used)\n* The authors should mention the shapes of the $M_est$ and $M_att$ and how they are matched.\n* What is the role of the softmax? Since it is cross-attention with the text tokens, aren’t the attention values normalized across the #text tokens already?\n\nResults\n---------\n* Can the authors confirm that the images apart from the 2.9K descriptions that are removed, all have color information?\n* The authors describe a multi-instance dataset. Are there plans to opensource this dataset?\n* In Figure 1, the authors describe three levels of colorization descriptions. The authors show some qualitiative results per-level, can they report some quantitative results as well?\n* Can the authors also report some of their failure cases, if any?\n\nOther\n-------\nFigure 2 combines both the training and sampling loop of the diffusion process, so it is quite confusing. I suggest that the authors separate both and clearly label the diagram wherever the loss is applicable.", "limitations": "I do not believe there is negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The results shown by their proposed approach are quite strong. The authors propose to augment a pretrained stable diffusion model to achieve language-conditoned colorizaiion which is novel and interesting. The authors provide adequate background information.", "weaknesses": "While the model works well, imho some novelty aspects are overclaimed and some simpler baselines are not ablated. I initially rate the paper slightly below borderline but I'm happy to update my rating it the authors address my concerns. See below for a detailed list of questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the very detailed review and suggestions.  \n\nGiven the character limit (6000), we have to make our response brief. For additional details, we welcome a more comprehensive discussion during the Author-Reviewer Discussions.\n\nNote Fig. S1-S13 and Tab. S1-S2 are included in the PDF attached to the global response.\n\n### ***Most important***\n- Q: How are these hyperparameters tuned?\n\nWe follow Stable Diffusion [31] to set loss hyperparameters $\\alpha=1.0$ and $\\beta=0.5$, and empirically set hyperparameters $\\lambda=0.1$ and $N_\\mathrm{win}=7$.  \n\nWe visualize artifact maps with varying $N_\\mathrm{win}$ to illustrate its practical role, as shown in Fig. S2. We further modulate $\\lambda$ and present qualitative and quantitative results in Fig. S9 and Tab. S1, respectively. This demonstrates that the value of $\\lambda$ is not sensitive to variations in a specific range.\n\n### ***Luminance-guided image compression***\n- Q: Explain the architecture of compression encoder/decoder and how luminance encoder features are incorporated.\n\nWe implement the compression encoder and compression decoder as the same structure of Stable Diffusion [1], as shown in Fig. S1. As presented in Fig. 2 (a), luminance encoder extracts multi-scale features from grayscale images (L133-135). Then, these features are added to corresponding scales of the compression decoder (L135-136).\n\n- Q: Explain discriminator loss in Sec. 3.2.\n\nWe adopt the exactly same discriminator loss as Stable Diffusion [1], therefore we do not perform an ablation study for it. \n\n- Q: Aablating this weighting factor in Eq. 3.\n\nWe apply an additional factor $\\gamma$ to adjust the weight of $\\mathcal{L}_\\mathrm{rec}$ when training our model in the pixel space. The qualitative and quantitative results are shown in Fig. S10 and Tab. S1, respectively. This demonstrates the value of $\\gamma$ is also not sensitive to variations in a certain range.\n\n### ***Semantic-aligned latent representation***\n- Q: Novelty overclaim in Sec. 3.3.\n\nThanks for pointing this out. Although CEC block is mathematically equivalent to using a stack of convolutions to extract features and add their output to the downsampling block, we present it this way to underscore the core motivation and practical value of the module in establishing a semantically-aligned latent representation (See L159-160 and L165-168). We will revise this in the final version to tone down the claim of novelty, *e.g.*, removing ''novel'' in L157 and explaining the mathematical equivalence.\n\n- Q: The concatenation baseline.\n\nWe conduct an additional ablation study ''concat''. Given the increase in the number of input channels, we extend the input channel of the first convolution layer. This modification leads to clear degradation in performance. We show the qualitative and quantitative results in Fig. S11 and Tab. S1, respectively. \n\n### ***Instance-aware sampling strategy***\n- Q: Segmentation guided baseline.\n\nWe build a ''segmentation'' baseline, where estimated object contours are concatenated with luminance features. The qualitative and quantitative results are shown in Fig. S11 and Tab. S1, respectively. This demonstrates that our instance-aware sampling strategy (ISS) offers a more effective control strategy.\n\n- Q: Which is semantic segmentation model used?\n\nWe exactly use SAM [18] as the segmentation model when sampling colorization results. Note that our model could switch to an arbitrary segmentation model without finetuning with ISS.\n\n- Q: Given shape of $M^\\mathrm{est}$ and $M^\\mathrm{att}$\n\nThe shape of $M^\\mathrm{att}$ is $\\bar{h} \\times \\bar{w} \\times N_\\mathrm{obj}$, where $\\bar{h}$ and $\\bar{w}$ are corresponding spatial resolution at $l$-th CA block, and $N_\\mathrm{obj}$ is the number of objects in the description. The shape of $M^\\mathrm{est}$ is $H \\times W \\times N_\\mathrm{obj}$, where $H$ and $W$ are spatial resolutions of input grayscale image. we downsample its spatial resolution to $\\bar{h} \\times \\bar{w}$ to ensure compatibility with $M^\\mathrm{att}$.\n\n- Q: The role of the softmax.\n\nWe apologize for typos in L193-194, which should be rewritten as:\n\nL193 $\\mathcal{M} \\leftarrow \\mathrm{Sigmoid}(M^\\mathrm{att}_l)$\n\nL194 $\\hat{M}^\\mathrm{att}_l \\leftarrow M^\\mathrm{att}_l - \\lambda \\nabla _\\mathcal{M} \\mathcal{L} _\\mathrm{BCE}(\\mathcal{M}, \\hat{M}^\\mathrm{est}_l)$\n\nWe integrate L193-194 into the attention mechanism. Specifically, we execute matrix multiplication to compute unnormalized attention maps, and then modify them (L193-194) followed by a softmax operation. This procedure aligns $M^\\mathrm{att}_l$ with downsampled estimated contours $\\hat{M}^\\mathrm{est}_l$. \n\nWe will revise this in the final version.\n\n### ***Result***\n- Q: Whether all data have color information.\n\nPrevious L-CoDe [41] and L-CoIns [5] ensure all descriptions of the extended COCO-stuff and multi-instance dataset include color information.\n\n- Q: Release Multi-instance dataset.\n\nL-CoIns [5] has released the multi-instance dataset.\n\n- Q: Per-level qualitative results.\n\nThere is no distinct criterion to partition descriptions of the extended COCO-stuff dataset and multi-instance datasets into complete-level and partial-level representations. As such, we present the quantitative results which include descriptions from both levels, as shown in Tab. 1 (left). We further show qualitative results with scarce-level descriptions in Tab. 1 (right) and Tab. 3 of the supplementary materials.\n\n- Q: Failure cases.\n\nGiven that the estimated object contours are utilized solely in the latent space and subsequently become low-resolution, our method still encounters challenges when attempting to accurately colorize small objects with corresponding color descriptions. We provide failure cases in Fig. S12.\n\n### ***Other***\n- Q: Confused Fig. 2.\n\nThanks for the helpful suggestion. We have redesigned Fig. S13 to serve as the revised Fig. 2, enhancing the clarity of our pipeline.\n\nWe will revise this in the final version.\n"}]

Author rebuttal:
We thank the reviewers for their insightful comments and acknowledging that the paper is well-written (Gu3m/Gyr2/whze), well-motivated (Gu3m), and easy to follow (Gyr2/whze); proposed method is effective (Gu3m), reasonable (Gyr2), interesting and can be generalized (zuAx), and novel and interesting (oFt4); authors conduct extensive ablation or even human evaluation experiments (Gu3m/Gyr2/whze/zuAx); results are advantaged (Gyr2), promising (whze), convincing (zuAx), and quite strong (oFt4); tackled problem is interesting and valuable (whze); and background information is adequate (oFt4). We have carefully considered your comments and will take them into account to further improve the quality of our work. Please find below our responses to specific concerns of each individual reviewer. 

Note that Fig. S1-S13 and Tab. S1-S2 can be found in the PDF attached to the global response. Once our paper is accepted, we will release the code and checkpoints to facilitate reproducibility and further research in this area.

We remain committed to addressing any further questions or concerns from the reviewers promptly.

Meta-review: All the reviewers recommend the acceptance of the work. Reviewers appreciated the well-motivated and well-engineered method with good results. Multiple reviewers raised concerns with respect to readability and clarity of several parts of the paper. Authors addressed several of these concerns in the rebuttal and reviewers are satisfied with their responses. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper, which include adding the relevant rebuttal discussions and revisions in the main paper. The authors are encouraged to make the necessary changes to the best of their ability.
Decision: Accept (spotlight)