SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Subsequently, we fix their weights and train the noise prediction network in the latent space using Eq. (6) for 50 epochs. The training process takes approximately 60 hours and is performed on 2 NVIDIA GeForce 3090Ti graphic cards. We employ the Adam optimizer with a learning rate of  $1 \times 10^{-6}$  for the pixel space and  $5 \times 10^{-5}$  for the latent space to minimize the losses. For sampling, we use the DDIM sampling method [38] with 50 sampling steps.
Dataset. We conduct our experiments on language-based colorization datasets:  $(i)$  the extended COCO-Stuff dataset [45], which is built upon the COCO-Stuff dataset [3] by discarding unqualified samples for the colorization task, including 59K training images and 2.4K evaluation images; and  $(ii)$  the multi-instance dataset [6], which provides samples featuring multiple instances with different visual characteristics within a single image, including 65K training images and 7K evaluation images. For both datasets, each image is accompanied by a corresponding language description.
4.1 Comparison with state-of-the-art methods
We make comparisons with language-based colorization methods (e.g., LBIE [7], ML2018 [27], Xie2018 [49], L-CoDe [45], L-CoDer [5], and L-CoIns [6]) and automatic colorization methods
Table 2: User study results. * marks our method receiving scarce-level descriptions. Ours (L-CAD) clearly produces a higher score than state-of-the-art methods on both datasets.
Corresponding experiment
Method	Extended COCO-Stuff	Multi-instance
LBIE	1.68%	3.20%
ML2018	3.28%	4.92%
Xie2018	0.96%	1.72%
L-CoDe	11.56%	11.32%
L-CoDer	16.32%	15.24%
L-CoIns	24.76%	19.56%
Ours	41.44%	44.04%
Ground truth	N/A	N/A
Reality experiment
Method	Extended COCO-Stuff	Multi-instance
CIC	6.24%	3.76%
InstColor	11.68%	10.36%
ChromaGAN	7.16%	6.32%
BigColor	11.48%	9.56%
DISCO	8.92%	10.84%
CT²	13.36%	12.92%
Ours*	19.72%	20.28%
Ground truth	21.44%	25.96%
(e.g., CIC [52], InstColor [39], ChromaGAN [40], BigColor [20], DISCO [47], and  $\mathrm{CT^2}$  [44]) to demonstrate that our model is capable of effectively handling any-level descriptions.
Qualitative comparisons. Previous language-based colorization methods assume that users provide comprehensive color descriptions, resulting in suboptimal performance, particularly for objects without corresponding color descriptions. In contrast, leveraging the prior knowledge of Stable Diffusion [34] and our newly proposed instance-aware sampling strategy, our L-CAD presents vivid colorization results with any-level descriptions, as shown in Fig. 3. we further make comparisons with automatic colorization methods using scarce-level descriptions that lack meaningful color information. As shown in Fig. 4, our model still produces superior colorization results.
Quantitative comparisons. Following previous works [5, 6, 45], we use Peak Signal-to-Noise Ratio (PSNR) [16], Structural Similarity Index Measure (SSIM) [42], and Learned Perceptual Image Patch Similarity (LPIPS) [53] to qualitatively evaluate the quality of colorization. With complete-level and partial-level descriptions, our method (L-CAD) outperforms all language-based colorization methods. Furthermore, with scarce-level descriptions, our method shows a distinct advantage over automatic colorization methods. As shown in Tab. 1, our method achieves the best PSNR, SSIM, and LPIPS scores on both datasets.
4.2 User study
We further conduct user studies to evaluate the subjective perception of human observers:  $(i)$  Corresponding experiment: We apply language-based colorization methods to colorize images with random complete-level or partial-level descriptions. Participants are instructed to select the image that best matches the description.  $(ii)$  Reality experiment: Participants are asked to select the image they perceive to be the most visually realistic among the real image, images colorized by our method with scarce-level descriptions, and those colorized by automatic colorization methods. For each experiment, we randomly select 100 samples from the testing images of each dataset, and conduct them independently by 25 volunteers on the Amazon Mechanical Turk (AMT). As shown in Tab. 2, our method achieves the highest scores in both experiments.
4.3 Ablation study and discussion
We create three baselines to study the impact of our proposed modules and the sampling strategy. The evaluation scores and colorization results of the ablation study are shown in Tab. 1 and Fig. 5.
W/o Luminance-guided Image Compression (LIC). We disable the luminance encoder in the pixel space and revert to using the perceptual compression model in [34]. Without the guidance of multi-scale grayscale features, the model cannot correctly preserve local spatial structures.
W/o Semantic-aligned Latent Representation (SLR). We remove luminance features that guide the semantic alignment in the latent space and therefore replace channel-extended convolution blocks with vanilla convolution blocks. As a result, the obvious ghosting effect occurs in colorized images.
Figure 5: Ablation study. When our proposed modules are disable, results become counterintuitive.
Figure 6: Compared with text-driven editing methods, our model colorizes images to be spatially aligned with grayscale images.
W/o Instance-aware Sampling Strategy (ISS). We replace the instance-aware sampling strategy with the standard DDIM [38], which significantly degrades the performance of the model to correctly assign colors to corresponding objects that have different descriptions.
Discussion. Recent studies (e.g., ControlNet [50], Pix2PixZero [30], and SDEdit [28]) could also edit images with descriptions, based on the pretrained generative models. However, these methods are not tailored to the colorization task, resulting in challenges related to preserving local spatial structures, leveraging rich color priors, and learning correspondence between objects and color words. These limitations make it difficult for them to perform instance-aware colorization with any-level descriptions. We present their unsuitability in Fig. 6.
4.4 Evaluation on ImageNet dataset
We conduct extensive experiments on the widely-used ImageNet dataset [36], including 1.3M images covering 1000 categories, to evaluate the performance of our L-CAD for automatic colorization. To provide a comprehensive evaluation, we introduce the Fréchet inception distance [12] to quantify the distribution gap between colorized images and ground truth images, as well as the colorfulness score [11] to reflect the vividness of colorized images. Following previous works [2, 22, 44], we evaluate our model on the first 5K images of the public validation set, where all the test images are center cropped and resized into  $256 \times 256$  resolution.
As shown in Tab. 3, our model achieves top scores in FID, PSNR, SSIM, and LPIPS metrics, and demonstrates comparable performance in colorfulness metrics. These results indicate that our model aims to achieve photo-realistic colorization, emphasizing natural tones rather than exaggerated saturation. It should be noted that due to the absence of provided descriptions in the ImageNet
Table 3: Quantitative comparison with more automatic colorization methods on ImageNet dataset. * marks our method receiving scarce-level descriptions.
Method	FID↓	PSNR↑	SSIM↑	LPIPS↓	colorful↑	△colorful↓
CIC [52]	8.72	22.64	0.91	0.22	31.60	4.72
DeOldify [1]	9.45	21.12	0.83	0.24	22.70	13.62
ChromaGAN [40]	7.66	23.35	0.90	0.21	27.88	8.43
InstColor [39]	8.06	23.28	0.91	0.21	24.87	11.44
GCP [46]	5.95	21.68	0.88	0.23	32.98	3.34
BigColor [20]	10.08	21.35	0.87	0.25	30.60	7.12
DSICO [47]	7.99	20.31	0.86	0.23	48.43	10.70
ColTran [22]	6.44	20.95	0.80	0.29	34.50	2.24
CT² [44]	5.51	23.50	0.92	0.19	38.48	2.17
ColorFormer [18]	4.64	23.14	0.89	0.18	37.95	0.23
L-CAD*	4.36	24.47	0.92	0.16	34.04	3.68
1940. "Japanese fruit farmer and his son. Placer County, California."
The two men wear khaki and blue shirts, and the curtains behind them were also golden.
1913. "Main Street, a commercial and residential block at the corner of Main and Cabot."
Figure 7: Examples of applying our method to colorize legacy photos with any-level descriptions.
There's a black car on the left and a cyan car on the right.
There's a blue car on the left and a red car on the right.
1906. "Scene at Roc Creek, zoo park, Washington, D.C."
A colorful image
dataset [36], our model is trained solely with scarce-level descriptions (e.g., "a colorful image") and leverage the rich image semantics encapsulated in language-based descriptions by initializing weights pretrained on the extended COCO-Stuff dataset [45]. This deviates from the training strategy adopted in language-based colorization datasets [6, 45] (detailed in Sec. 3.5), potentially affecting the performance of our model.
4.5 Application
We apply our method to colorize legacy black-and-white photos using descriptions at the complete, partial, and scarce levels, respectively. Results are presented sequentially from left to right in Fig. 7, which demonstrates the robust generalization capability of our model.
We present L-CAD, a unified model to perform Language-based Colorization with Any-level Descriptions. Leveraging the prior knowledge of the pretrained model, we design novel modules to preserve local spatial structures and prevent the ghosting effect. We further propose the instance-aware sampling strategy to colorize diverse and complex scenarios. We demonstrate our model could handle any-level descriptions by making comparisons with language-based colorization methods with complete-level and partial-level descriptions, as well as automatic colorization methods with scare-level descriptions. Both qualitative and quantitative results demonstrate our superior performance.
Limitations. Our model as a diffusion model takes an additional sampling process for visualizing colorization results. This requirement considerably slows down the generation speed, and further limits the potential for real-time applications of our model. To address this concern, adopting advanced fast sampling methods (e.g., DPM-Solver [26]) could be a promising direction.
Acknowledgement. This work is supported by the National Key R&D Program of China under Grant No. 2021ZD0109800, the National Natural Science Foundation of China under Grand No. 62136001, 62088102, and Program for Youth Innovative Research Team of BUPT No. 2023QNTD02. Yu Li is supported by Shenzhen Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone, under Grant No. HTHZQSWS-KCCYB-2023052.
[51] L. Zhang, Q. Chen, B. Hu, and S. Jiang. Text-guided neural image inpainting. In ACM MM, 2020.
[52] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In ECCV, 2016.
[53] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[54] Y. Zhang, N. Huang, F. Tang, H. Huang, C. Ma, W. Dong, and C. Xu. Inversion-based creativity transfer with diffusion models. In CVPR, 2023.
[55] J. Zhao, J. Han, L. Shao, and C. G. Snoek. Pixelated semantic colorization. IJCV, 2020.
[56] J. Zhao, L. Liu, C. G. Snoek, J. Han, and L. Shao. Pixel-level semantics guided image colorization. In BMVC, 2018.