Title: SSIMBaD: Sigma Scaling with SSIM-Guided Balanced Diffusion for AnimeFace Colorization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Models

Noise Scheduling

Perceptual Consistency

Reference-guided Generation

Conditional Diffusion Models

Generative Modeling

Sketch-to-Image Translation

Structural Similarity Index
Abstract: We propose a novel diffusion-based framework for automatic colorization of Anime-style facial sketches, which preserves the structural fidelity of the input sketch while effectively transferring stylistic attributes from a reference image. Our approach builds upon recent continuous-time diffusion models, but departs from traditional methods that rely on predefined noise schedules, which often fail to maintain perceptual consistency across the generative trajectory. To address this, we introduce SSIMBaD (Sigma Scaling with SSIM-Guided Balanced Diffusion), a sigma-space transformation that ensures linear alignment of perceptual degradation, as measured by structural similarity. This perceptual scaling enforces uniform visual difficulty across timesteps, enabling more balanced and faithful reconstructions. On a large-scale Anime face dataset, SSIMBaD attains state-of-the-art structural fidelity and strong perceptual quality, with robust generalization to diverse styles and structural variations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a EDM-based method, namely SSIMBaD, for anime face colorization. The authors believe that a diffusion process that allows the SSIM metric to degrade linearly over time is conducive to training a model with good generative capabilities. To realize this objective, SSIMBaD introduced a $\\sigma$ transformation to noise schedule and achieved symmetry in the training process and testing process. Qualitative and quantitative experiments show that SSIMBaD outperforms the compared methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is organized well  and easy to understand.\n2. The performance of the proposed SSIMBaD is better than the compared methods.\n3. The paper is easy to reproduce and the code is provided.\n\nWeakness:\n1. The authors claim that linear degradation in diffusion process with SSIM metrics can improve the model’s capacity, but there is not enough corresponding experiments to demonstrate this claim. Specifically, the strategy in e.q. (6)  to select the $\\sigma$ transformation only show the coefficient of the linear relationship but not reflect the final metrics, which is not able to prove the superiority of the selected function.\n2. Lack of exemplar-based style transfer methods for comparisons. The anime face colorization problem can be regard a special cases of style transfer. Besides, SSIMBaD requires 2 H100 for training and training-free methods [1-3] only require one for inference. Therefore, it is necessary to compare these methods.\n3. The results presented in qualitative and quantitative experiments are not convincing enough. \n\nQualitative analysis: In addition to the eye color defects already pointed out in the article, the hair in the results generated in Figures 1 and 2 also lost the ‘reflective’ characteristics of the reference image. Another serious issue is that, as shown in the second row of Figure 2, there is a noticeable difference in hair color between the generated image and the reference image, and the reference image does not seem to serve as a guide for the generation.\n\nQuantitation analysis: Although the metrics of SSIMBaD in Table 2 are superior to other methods, they are still at a relatively low level, such as PSNR of 18.92 and 15.84. This may limit the practical application of the model. In addition, there is a lack of corresponding style metrics (e.g., DINO-I) to reflect the style differences between the generated images and the reference images.\n4. Minor weakness: \n(a) Lines 158-159 and e.q. (6) are missing punctuation marks.\n(b) The sub-figures (d-f) had better be resize to the same resolution.\n\n[1] Alaluf, Yuval, et al. \"Cross-image attention for zero-shot appearance transfer.\" ACM SIGGRAPH 2024 Conference Papers. 2024.\n[2] Zhou, Yang, et al. \"Attention distillation: A unified approach to visual characteristics transfer.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n[3] Hertz, Amir, et al. \"Style aligned image generation via shared attention.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\nSee weaknesses and the followings:\n1. Why choose SSIM as a metric for measuring image degradation rather than other metrics such as CLIP-I or FID?\n2. The author emphasizes linear degradation. Why not consider rectified flow instead of using EDM?\n3. Can authors provide the distribution of timesteps after $\\sigma$ transformation?\n\nLIMITATIONS:\nAlthough the author explains that SSIMBaD has limitations in handling eye colors, further detailed discussion of these limitations is possible. For example, why has the proposed method not been applied to broader example-based style transfer tasks?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SSIMBaD, a reference-guided diffusion framework for anime-style face colorization. The core idea is to replace EDM’s log/ρ noise schedule with a “perceptually aligned” σ-schedule obtained through a hand-picked transformation ϕ*(σ)=σ/(σ+0.3) that yields (approximately) linear SSIM degradation over timesteps. The same schedule is used in training, inference, and a lightweight reverse-only “trajectory-refinement” stage. Experiments on the 256² Danbooru AnimeFace dataset show large PSNR/MS-SSIM/FID improvements over SCFT and the AnimeDiffusion baseline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Clear motivation: The mismatch between perceptual difficulty and conventional σ-schedules in colorization is a real issue; the paper articulates it clearly.\n2. Simple, reproducible technique: The proposed rescaling is easy to implement inside any EDM-style pipeline.\n3. Comprehensive ablations: Table 3 isolates the influence of EDM, σ-scaling, and refinement.\n\nWeaknesses\n1. Missing strong baselines – No comparison with modern controllable diffusion systems (ControlNet[1], T2I-Adapter[2], IP-Adapter[3]) that can also ingest sketches + references.\n\n2.  Backbone diversity lacking – SSIMBaD is tested only with a U-Net. No evidence that the SSIM-aligned schedule benefits modern transformer backbones such as DiT[4].\n\n[1]: Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. \"Adding conditional control to text-to-image diffusion models.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n[2]: Mou, Chong, et al. \"T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 38. No. 5. 2024.\n[3]: Ye, Hu, et al. \"Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.\" arXiv preprint arXiv:2308.06721 (2023).\n[4]: Peebles, William, and Saining Xie. \"Scalable diffusion models with transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\nQUESTIONS:\n1. The paper currently compares only against SCFT and AnimeDiffusion. Could you add quantitative results for the more recent controllable-diffusion methods mentioned above (e.g., ControlNet, IP-Adapter, T2I-Adapter) so that the reported gains are positioned against the current state of the art?\n\n2. Have you evaluated the proposed SSIM-aligned σ-schedule with a transformer-based denoiser such as DiT (Peebles & Xie, 2022)? If not, please discuss whether you would expect similar improvements and provide any preliminary evidence or rationale.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SSIM-guided noise schedulers to enforce uniform visual difficulty across timesteps. The proposed method achieves state-of-the-art performance compared to baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\nPaper Strengths:\n1. It is reasonable to use SSIM-guided noise schedulers to ensure the uniform visual difficulty across timesteps.\n2. The paper is well-written and easy to follow\n3. The paper conducts extensive experiments to validate the effectiveness of the proposed method.\n4. The proposed method outperforms baseline methods.\n\nPaper Weaknesses:\n1. The authors need to further clarify the setting of the same-reference scenarios. In Line 200, it is said that the reference image is a perturbed version of the ground-truth. But from Figure 1, the structure is almost the same as the Sketch input. What kind of perturbations are applied to images?\n2. The proposed method should be compared with more methods, i.e., merely comparing with two methods is not enough.\n3. In Table 2, what are the additional 10 epochs used for? What’s the difference between the pretrained one and the finetuned one?\n4. How does SSIMBaD perform if 10 epochs are finetuned on top of SSIMBaD (w/o trajectory refinement)?\n\nQUESTIONS:\n1. The authors need to further clarify the setting of the same-reference scenarios. In Line 200, it is said that the reference image is a perturbed version of the ground-truth. But from Figure 1, the structure is almost the same as the Sketch input. What kind of perturbations are applied to images?\n2. The proposed method should be compared with more methods, i.e., merely comparing with two methods is not enough.\n3. In Table 2, what are the additional 10 epochs used for? What’s the difference between the pretrained one and the finetuned one?\n4. How does SSIMBaD perform if 10 epochs are finetuned on top of SSIMBaD (w/o trajectory refinement)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Sigma Scaling with SSIM-Guided Balanced Diffusion (SSIMBaD) for AnimeFace colorization. The key idea is to apply a sigma-space transformation to achieve a linear alignment with the Structural Similarity Index (SSIM), improving consistency during the denoising process across different timesteps. The authors explore several transformation candidates and empirically select the most effective one. Additionally, they introduce a trajectory refinement to enhance perceptual quality. Experimental results demonstrate that SSIMBaD achieves state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n1.  The sigma-space transformation is a novel and interesting idea that improves consistency in the denoising process. Moreover, experiments support the effectiveness of this idea.\n2. The experiments in this paper are overall comprehensive. The authors provide ablation studies and empirically justify their choice of transformation.\n\n**Weaknesses**\n1. The ablation study does not report performance for the setting with trajectory refinement but *without* SSIM-aligned sigma-space scaling. In Table 3, trajectory refinement contributes the largest performance gain. This result raises my question of whether trajectory refinement is actually the primary contributor. Moreover, this paper does not discuss how trajectories change before and after refinement.\n\n2. This paper’s presentation could be improved. The motivation in the introduction is unclear, and several abbreviations or notations are not defined in the main text. For example,  \"TPS\" is first mentioned in Line 138 but only defined in the appendix (Lines 404–405). Similarly, \"XDoG\" appears in Line 139 without citation or in-text definition, and is only explained in the appendix (Line 399).  Moreover, in Line 148, the terms $P\\text{mean}$ and $P^2\\text{std}$ are used without prior definition.\n\nQUESTIONS:\n1. What is the performance when using trajectory refinement without SSIM-aligned sigma-space scaling?\n2. In Table 4 of the appendix, the caption mentions TPS, but the table content is about trajectory refinement. Is this an error?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}