SketchEvo: Leveraging Drawing Dynamics for Enhanced Image Synthesis
Authors: Zhixin Feng
Runan Yin
LAN YANG
Kaiyue Pang
Ke Li
Honggang Zhang
Yi-Zhe Song
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion model; Image generation; Sequence Guided; Human Preference
Primary area: generative models
TL;DR: SketchEvo generates images aligned with human preferences by modeling the evolution of a sketch from the initial strokes to the completed drawing.

Abstract: Sketching represents humanity's most intuitive form of visual expression -- a universal language that transcends barriers. Although recent diffusion models integrate sketches with text, they often regard the complete sketch merely as a static visual constraint, neglecting the human preference information inherently conveyed during the dynamic sketching process.This oversight leads to images that, despite technical adherence to sketches, fail to align with human aesthetic expectations. Our framework, SketchEvo, harnesses the dynamic evolution of sketches by capturing the progression from initial strokes to completed drawing. Current preference alignment techniques struggle with sketch-guided generation because the dual constraints of text and sketch create insufficiently different latent samples when using noise perturbations alone. SketchEvo addresses this through two complementary innovations: first, by leveraging sketches at different completion stages to create meaningfully divergent samples for effective aesthetic learning during training; second, through a sequence-guided rollback mechanism that applies these learned preferences during inference by balancing textual semantics with structural guidance. Extensive experiments demonstrate that these complementary approaches enable SketchEvo to deliver improved aesthetic quality while maintaining sketch fidelity, successfully generalizing to incomplete and abstract sketches throughout the drawing process.

Published as a conference paper at ICLR 2026
SKETCHEVO: LEVERAGING DRAWING DYNAMICS FOR ENHANCED IMAGE S SYNTHESIS
Zhixin Feng', Runan Yin', Lan Yang Kaiyue Pang², Ke Li Honggang Zhang', Yi-Zhe Song? Beijing University of Posts and Telccommunications, China SketchX, CVSSP, University of Surrey, United Kingdom Chenxi Shuzhi (Beijing) Technology Co., Ltd. China
{fengzhixin, 15735838880, ylan, likel990 zhhg) edu, cn kaiyue, pangl 9936 3ma3mil...... com y. song@surrey, ac.uk uk
ABSTRACT
Sketching represents humanity's 's most intuitive form of visual expression - a uni- versal language that transcends barriers. Although recent diffusion models inte- grate sketches with text, they often regard the complete sketch merely as a static visual constraint, neglecting the human preference information inherently con- veyed during the dynamic sketching process. This oversight leads to images that, despite technical adherence sketches, fail to align with human aesthetic expec- tations. Our framework, SketchEvo, harnesses the dynamic evolution of sketches by capturing the progression from initial strokes completed drawing. Current preference alignment techniques struggle with sketch-guided generation because the dual constraints of text and sketch create insufficiently different latent sam- ples when using noise perturbations alone. SketchEvo addresses this through two complementary innovations: first, by leveraging sketches different completion stages to create meaningfully divergent samples for effective aesthetic learning during training; second, through sequence-guide rollback mechanism that ap- plies these learned preferences during inference by balancing textual semantics with structural guidance. Extensive experiments demonstrate that these comple- mentary approaches enable SketchEvo deliver improved aesthetic quality while maintaining sketch fidelity, successfully generalizing to incomplete and abstract sketches throughout the drawing process.
INTRODUCTION
Sketching is one of humanity's 's most intuitive forms visual communication. The act of drawing translating mental concepts into physical strokes naturally encodes rich information not just in the final result, but in the sequential process itself. Recent advances in diffusion models (Ho et al.) 2020 (oog al. 2021a Hu et al. 2024 [Sng et al., (021b) have enabled remarkable progress in controllable image generation, with approaches like ControlNet (Zhang et al.) (023), T21-Adapter (Mou al. 2024), and VersaGen (Chen et al. 2025) successfully integrating sketch conditions with
textual prompts.
However, these methods (He al.) 2024 Liu 2024 Li al. 2024] Qin al. 2023b Hu al.,
2023) primarily focus on the final sketch as static spatial constraint, overlooking the intermediate information of the drawing process. When processing amateur sketches, existing approaches often produce technically correct but aesthetically disappointing results - they satisfy structural constraints (often poor as per amateur sketches) yet fail to capture human intent. This disconnect reveals deeper problem: existing models don't understand how humans conceptualize and refine visual ideas through the progressive accumulation strokes.
Our investigation reveals that the core challenge lies in fundamental misalignment between gen- erated outputs and human aesthetic preferences. While recent preference alignment approaches like
"oorrespnndence like 9900 bupt.edu.cn Code to be found GitHub page.

---

Published as a conference paper at ICLR 2026
Hard drown onlie
Thrre ginafees forest.
konge
Book
Hond crown oréme style portrolt.
Kires flying together in The sky.
Chardeler:
cheese
style
way
bottle of Aire sitting on toble glass.
Step boand the field near
Comera
Hard drown onima style portront.
dY
Quick Draw
Sketchy
AnimeDiflusion
ESCOCO
Figure 1: Visualization of SketchEvo results.
The model produces images aligned with human
preferences, trained only on the Sketchy dataset.
DPOK Fan al. 2023). DPO (Rafailov al.) 2023), D3PO (Yang et al., 2024), SPO (Liang et al. 2025), and LPO (Zhang et al. 2025) have improved image quality in other contexts, they face critical obstacle in sketch-guided generation. These methods optimize models by comparing generated variations and adjusting to produce more preferred outputs, but in multimodal sketch- and-text generation, the dual constraints create insufficiently different samples when using conven- tional noise-based variations. These limited variations provide weak training signals that struggle to capture meaningful aesthetic improvements while preserving sketch details. This creates a false dichotomy where models must choose between faithfully reproducing potentially flawed amateur sketches or generating visually pleasing but cxetch-inconsisse images.
To address this challenge, we introduce SketchEvo, framework that leverages the sketch sequences from initial strokes to completion as a powerful source of diversity for preference based opti- mization. Our key insight is that intermediate sketches from different drawing stages represent vary- ing levels abstraction and detail, offering meaningful semantic and structural divergence while maintaining connection the user's intent. By using these sketch variations instead of relying solely on noise perturbations, SketchEvo creates meaningfully different sample pairs that provide strong signals for human preference aligmment even under tight multimodal constraints.
SketchEvo introduces two complementary innovations that work together across the model lifecycle: First, sequence-guide sampling strategy transforms sketching information into effective learning signals during training by incorporating sketches at different completion stages conditional inputs. This expands candidate diversity and creates sample pairs with greater aesthetic divergence, provid- ing more informative gradients for preference optimization. Second, building upon this improved preference alignment, our requence-guided rollback mechanism applies these learned preferences during inference by leveraging initial sketch strokes to guide rollback. This quantifies information gain from both textual and sketch conditions, ensuring the aesthetic improvements learned during training fully realized in the generated images while maintaining structural fidelity to the user's sketch intent.
Our extensive experiments demonstrate substantial improvements across multiple human preference metrics and conditional fidelity measures. Particularly noteworthy is SketchEvo' 's strong generaliza- tion ability in on-the-fly sketch-to-image tests, where successfully produces high-quality images
2

---

Published as a conference paper at ICLR 2026
from incomplete and abstract sketches various stages of the drawing process, enabling truly inter- active sketch-based creation.
2
RELATED WORK
2.1
CONTROLLABLE IMAGE GENERATION
The rapid development of diffusion models has significantly advanced text-to-image genera- tion (Podell al. 2024 Esser et al. 2024 Labs 2024). Building on pretrained T2I models, various conditional strategies Ye et al. [023] Ruiz et al. 2023] Chen et al., 2024: Wang et al.) (224) in- corporate external structural cues, with sketches serving as an intuitive prior. Early approaches such as ControlNet (Zhang et al. 2023) and T2I-Adapter (Mou et al., 2024 introduce sketch features via auxiliary branches, and ControlNet++ (Li et al. 2024) further refines spatial consistency. Subse- quent extensions including VersaGen (Chen et al. 2025). SmartControl (Liu et al. 2024). and Uni-
Control (Qin et al. 2023a) -ehance robustness by expanding controllable modalities or improving feature injection. AnimateDiff (Guo et al. 2024) extends such conditioning into temporal domains, highlighting the utility of structured priors beyond single images. (Koley et al. 2024) improves ro- bustness when sketches are rough, imprecise, or drawn by amateurs, while KnobGen (Navard al.) 2024) tackles sparse and structurally incomplete sketches through a dual-path design that jointly models coarse and fine structural cues. However, most existing approaches treat sketches merely as static inputs, overlooking the rich intermediate information and implicit human preference signals embedded in the drawing process.
2.2
HUMAN PREFERENCE OPTIMIZATION
Various aesthetic metrics have been proposed, such as ImageReward Xuu et al.) 2023). HPSv2 (Wu al. 2023), PickScore (Kirstain et al. (023), and Reward (Xu al. 2024)) Re-
cent approaches improve subjective alignment through rollback mechanisms (LiChen et al. 2025
RLHF-inspired formulations (Ouyang et al. 2022] Zhang et al.) 2024 Fan al.) [023 Black al., Rafailov et 2023 Yang et al. 20244) and preference optimization with sample pairs (Liang
2024
et al. 2025 Zhang al. 2005] Yeh et al. 2024). However, their reliance on random Gaussian noise
perturbations limits the capture of structural and layout-level aesthetic intentions.
PROBLEM STATEMENT AND BACKGROUND
Problem formulation. This paper investigates the aesthetic degradation problem in sketch- guided image generation, with inputs being textual descriptions and a entire drawing sequence {s1, $2..... 8x) from the first stroke 8 to the completed sketch sx. The goal is to generate an image that combines high aesthetic quality and sketch fidelity. However, current human prefer- ence alignment methods Fan et al. (2023); Wallace et al. (2024); Yang et al. 20 Liang et al. (2025); RRaailov et al. (2023) face optimization bottlenecks in multimodal generation tasks with dual constraints of text and sketch. The core issue lies in their reliance single Gaussian noise perturbation, leading to insufficient diversity in generated sample pairs only subtle differences can be produced under dual constraints, making effective optimization diffieult.
Preference Optimization for Diffusion Models. Aligning diffusion models with human aesthetic preference is very challenging. Given a winning image x." losing image x', and text condition c, methods such as DPCRafailov et al. (2023) and D3PC et al. (2024) generate intermediate latent image pairs (xt" .z)). thereby incentivizing the diffusion model pe to prioritize the generation of x over z,. However, the preference ordering align with the initial preferences (x=", 2²)) To address this issue, SPO Liang et al. (2025) tackles by ), and introduces a Step-Aware Preference Model (SPM) to predict preference scores for samples the candidate pool, thereby constructing refined preference pairs (f)) The optimization objective of SPO is:
during the denoising process does not always
sampling from x++1 to generate denoised sample candidate pool (xt,,
LSPO -EgY :
[log (34rl)
po(21" 1it1 ((+1,1(+1) Po (xt (4.1c11+1 (rt
Ar
2/ ( 1) (aat(ar o c(+.ct+1)
where the symbol a denotes the sigmoid function, 3 is a regularization hyperparameter and Pref denotes the reference value from the fixed initial denoising model po-
3

---

Published as a conference paper at ICLR 2026
Construction of Denoised Sample Candidate Pool. SPO constructs the denoised sample candidate pool (z). 2, by adding random Gaussian noise & on the generated samples μıر (T C, ++ 1), as formalized below:
where (O, T), denotes the k-th sample in the candidate pool. Sample diversity of candidate pool is solely governed by the noise 2 sampled from a standard Gaussian distribution.
4 PROPOSED METHOD
swan glies on the lake, white faathers rippling
LoRA F Fineluned Diffusion Model
DM
Sy
Nee
LoRA
Score model
LoRA DM
X
SGPO
LoRA DM
random
SGPO
Xy
x
Training Phase
CSN
SGR
SGR
glides
Xte1
SGR
81+1
Xo
foothers rippling
Inference Phase
Figure 2: The framework of SketchEvo. During training, the SGPO module can obtain distinctive positive-negative sample pairs, fine-tuning the LoRA model in the U-Net align with human pref- erences. During inference, a SGR mechanism is employed to strengthen conditional information.
4.1 SEQUENCE-GUIDEE PREFERENCE OPTIMIZATION (SGPO)
Sequence-Guided Sampling Strategy.
Unlike traditional methods that solely rely on random noise perturbations generate candidate samples, we introduce sketch sequences 82..,,.... &N) from the drawing process to construct dynamically evolving candidate pool. Specifically, as shown in Fig. 2 at each sampling step, K sketches are randomly selected from the sketch sequence conditions to guide the model in through denoising 11+1- This process is formally
constructing a candidate pool {at, xē,......
expressed as:
uol C, +11+++2
where ch denotes the sketch condition 8, adopted for the k-th sample aft
Within the dynamically evolving candidate pool, differences between samples are no longer entirely dominated by random noise z. We leverage sketches at different completion stages as conditional puts to guide the denoising process, thereby generating diversified denoised samples. Since sketches at each intermediate stage undergo significant evolution structural and detail levels (As shown in Fig. 2 within the swan sketch drawing sequence, substantial discrepancies exist between 81, Sn, 8x), this approach enables the generation diversified samples.
Positive-Negative Sample Pair Selection Strategy. At each diffusion timestep t, utilize pretrained scoring model to evaluate the samples in the dynamic evolving candidate pool. The highest- and lowest-scoring samples selected and denoted po(" 2 Cr,, 1) and

---

Published as a conference paper at ICLR 2026
po(at |z/11: C, Ca,, 1), respectively, forming positive-negative sample pair. To better optimize the model, only sample pairs surpassing predefined threshold are utilized for model training.
The Impact of Candidate Pool Sample Diversity on Preference Optimization. According to equation given two samples oy and z the gradient of the reward function with respect to the SGPO, Eq. is reformulated follows:
VeLoure - (3(-)) (Ve log po(2 Caw 1 1....,ttt at Velog 2211, c, 1))]
(5)
Po po ait1, c, Can. t 1)
po(at c,,,,
Ar
(6)
Pret(x"" 1x41, C, Cau , 1)
Pres(ark 1441, C, Cap
where C Caj represent the sketches condition corresponding to the best latent and the worst latent, respectively.
As derived from Eq. 5 the effectiveness of gradients depends the discrepancy between æt and r/. When sample diversity is insufficient, this discrepancy diminishes, causing Ar - 1, which de- generates gradients into an uninformative signal with poor directionality. It can be observed that our proposed SGPO significantly outperforms SPO in candidate pool sample diversity. This disparity is directly reflected in the differences between sample pairs SGGP generates positive-negative sam- ple pairs with more pronounced disparities. The higher sample diversity enables SGPO to provide richer gradient information for the model during the optimization process.
4.2 SEQUENCE- GUIDDD ROLLBACK (SGR) MECHANISM DURING INFERENCE PHASE
To effectively transfer the improvements of SGPO from the training phase to the inference phase, we introduce a sequence-guided rollback mechanism. Previous works have demonstrated that in- corporating a rollback mechanism into the text-guided image generation process allows the latent encoding to more effectively capture semantic information. The cumulative latent difference in roll- back can be formulated as: where h, = vI/a, - - vI/an_1- - -I, is the model-predicted noise, Ti (t) denotes the semantic information gain term and (2(t) denotes the error term, which can be neglected under certain conditions.
When directly transferring existing rollback mechanism sketch-to-image generation tasks, their performance improvement faces significant bottlenecks. The fundamental reason lies in the fact that methods design optimization objectives solely for uni-modal textual conditions, failing model the structural priors inherent in sketch conditions. To address this issue, we propose the SGR mech- anism, which implemented by integrating the sketch-drawing sequence and text conditions to jointly guide the rollback process, formulated below:
<x,,,:) (1 + (1)ou((u Cex 71 7ou 0, CaN () (.x)) + t2 C, Cen, 72u 0, Can.
In our method, the information gain encompasses not only semantic information from text condi- tions but also structural and detailed information introduced by the sketch sequences 8a, as well as human preference information learned by the model parameters During the inference phase, when the sketch sequence 5, is predetermined, our rollback mechanism is simplified to follow the generation manner text-to-image. Consequently, we adopt the configuration of 71 and 72 to max- imize semantic information enhancement, With 71 and 72 fixed, the information gain is determined by the following formula (detailed proofs are provided in Appendix D):
T òz Öa-Somplnng O [(n(t)) & (us((a Can, t))²
(8)
The discrepancy in structural and detailed information gain is determined by the noise generated through the control of textual conditions and sketch conditions Cs,, in the generative model. By increasing the divergence between e and Cs,. the cumulative information gain can be augmented.
The SGR mechanism quantifies the information gain from text and sketch conditions, ensuring that aesthetic enhancements learned during training are fully manifested in the generated images while preserving structural fidelity to the user's original sketch intent.
5

---

Published as a conference paper at ICLR 2026
Table I: Quantitative results. We evaluate our method and competitors on the sketchy dataset in
terms of human preference alignment and conditional fidelity.
Human Preference Alignment
Conditional Fidelity
Method
Image Reward
HPS v2
Pick
LPIPS-sketch + CLIP-Scure
ControlNet ZZhngg
2023
0.004
24.08
20.03
0.11
23.70
T2i-Adapter Moou ar. 2024
00,01
23.60
20.58
0.20
15.86
VersaGen CChen 2025
0.08
24.68
20.79
0.14
23.77
ControlNe [l. 2074
24.19
20.56
AnimateDiff Guo
2024
0.23
23.68
20.42
0.14
23.56
UniControl Qin cf 2445B
0.024
23.08
20.39
0.13
23.75
ControlNet- DPC [Kaallo ct 2023
0.47
25.02
20.86
0.16
23.35
SPrnolNee-SPO((
2025
27.69
22.04
0.17
23.65
SGPO
28.87
21.94
23.86
Ours
1.18
30,08
22.41
0.15
24.15
EXPERIMENT
5.1
EXPERIMENTAL SETUPS
Our method is fine-tuned based on the ConLrolNetXL (Podell et al. 2024 to achieve alignment with human preferences. All experiments are conducted on NVIDIA ATO0 GPUs for both training only with Sketchy (Sangkloy et al.) 2016) dataset Detailed information about data processing, hyperparameters employed the experiments are provided in the Appendix A and
5.2 COMPARISON WITH THE STATE-OF-THEEART METHODS
To verify the effectiveness our method, we compared it with state-of-the-ar multi-condition con- trolled image generation models ControlNet (Zhang et al. 2023), T2i-Adapter (Mou et al. 2024
VersaGen (Chen al. 2025), ControlNet++ (Li al. 2024). AnimateDiff (Guo al. 2024).
Unicontrol Qin et al. (2023a) and preference aligning methods DPO (Rafailov et al. 2023) and
SPO Liang et al. 2025) under their original settings. Since the original DPO and SPO are based on SDXL (Podell et al. 2024 fine-tuning and only accept text conditions, replaced the SDXL model in ControlNet with their fine-tuned SDXL models. SGPO refers to our proposed method without employing the rollback mechanism.
For Human Preference Alignment, as shown in Tab. We found that, compared with multi- condition controlled image generation models, methods after preference alignment achieved aes- thetic score improvements, with ours outperforming ControlNet-DPeS and ControlNet-SPO From a detailed view, human preference is reflected in three aspects: i) Position: As shown in the first row of Fig. 3 The cat and chair generated by our model exhibit a more natural positional relationship. demonstrating superior spatial layout preference capability. ii) Composition: As presented in the second row of Fig. 3, the cake generated by our model exhibit proportionally more realistic scaling relative to the candles, demonstrating superior compositional preference capability. iii) Color: illustrated in the third row of Fig. 3 our model accurately identifies the color boundaries between foreground and background, altering only the helicopter's main body camouflage pattern while preserving the background's original appearance, demonstrating precise control over color prefer- ences.
Table Quantization results demonstrating the model's generalization capability when trained only
on the Sketchy dataset and evaluated on unseen data.
Image
LPIPS Sketch
CL.IP
Dataset
Method
Reward
HPS v2
PickScore
Score f
ControlNet
-0.56
15.24
17.17
0.47
23.07
QuickDraw
ControlNet-SPO
0.40
27.05
21.38
24.01
Ours
0.86
30.22
21.67
0.68
24.28
ControlNet
-0.11
18.34
19.62
0.10
23.17
AnimeDiffusion
ConSrolNet-SPO
0.27
23.28
19.67
0.15
23.99
Ours
1.32
31.57
23.68
0.10
24.96
ControlNet
-0.03
23.13
18.99
0.41
24.05
FSCOCO
ConOrolNet-SPO
0.52
27.51
19.09
0.50
24.39
Ours
0.96
30.31
21.78
0.45
24.77

---

Published as a conference paper at ICLR 2026
Cantroliad
12-4
Vamelien
Controciat-
EurContral
Controliie 4PO Contrehhe-SUP SUP
Foncy with cardie.
hellopphrr.
Taix honass
Gonal
Broad
Figure 3: Visualized comparison of ControlNet, T21-Adapter VersaGen, AnimateDiff, Control- Net++, UniControl, ControlNet-DPO, ControlNet-SPP SGPO and Ours on Sketchy dataset.
CostrreNio
Coumseleer
Ceenwl
Ner
Nor
plata full donutr.
Rock Rog.
Hend draan reapp style portnot.
Sind sitting slick
Anriclancce
Hund drown anline style partrolt.
Yolo grazing in the fichl
Cokee
Hand dream
hydrannt side reed
onime city
Alg vasi afrectt
QuickDeaw
Figure 4: Generalization results are visualized, consistently generating high-quality images regard- less of sketch expertise. Images from professional sketches show even better alignment with inputs.
As shown in Tab. [ method achieves the highest semantic fidelity, though the highest sketch similarity, as ControlNet and T21-Adapter strictly follow sketches. Compared to preference- alignment methods like DPO and SPO, our sequence-guided approach better integrates text and sketch information, preserving fine-grained structures while generating natural images. As illus- trated in Fig. 3 (rows 4-6), simultancously: (i) accurately realizes quantitative semantics (c.g.. two bananas); (ii) suppresses redundant strokes (c.g., camel head) via enhanced semantic supervision; (iii) enhances visual expressiveness while maintaining precise text-image consistency.
5.3 GENERALIZATION ABILITY OF THE MODEL
To further evaluate the generalization ability of our model, conduct inference the Quick- Draw (qui) 2016). AnimeDiffusion (Cao et al. 2024), and FSCOCO (Chowdhury et al. 2022) datasets. As shown in Tab. 2 and Fig. 4 our model consistently achieves competitive performance

---

Published as a conference paper at ICLR 2026
across diverse datasets, demonstrating robust generalization beyond the training domain. More sults demonstrations are presented in G
For single-object sketch datasets, the model adaptively balances aesthetic quality and sketch simi- larity according to the level of sketch abstraction, without requiring manual weight adjustment. As shown in Tab. 2 whether tested the abstract QuickDraw dataset the professional AnimeDiffu- sion dataset, our model consistently produces results with the highest aesthetic scores while main- taining strong sketch fidelity. For example, on the QuickDraw dataset, ControlNet generates flipped hot tub images that defy real-world structure, while ControlNet-SPO yields higher aesthetic scores but poor sketch alignment. contrast, our model automatically integrates both sketch consistency and aesthetic quality, generating superior images without manual tuning.
For complex scene sketch datasets, our model consistently achieves higher aesthetic scores and sketch similarity. As shown in Tab. 2 on FSCOCO, our method outperforms ControlNet-SPOO in both metrics. Due to the presence of multiple elements in scene sketches, overall similarity scores are lower, even ControlNet achieves only 0,41, underscoring the difficulty of strict alignment in complex scenarios.
Moreover, higher sketch professionalism leads to greater image-sketch similarity.As illustrated in Fig. and Fig. [ the similarity between images and sketches progressively improves from the less professional QuickDraw, through sketch and FSCOCO, to the highly professional Animeduffusion dataset. Notably, this enhancement in similarity is achieved without compromising aesthetic quality, demonstrating strong preservation of sketch details.
5.4 ABLATION STUDY AND ANALYSIS
Can Requence-Guider Sampling Strategy enhance the sample diversity of the candidate pool? Indeed, our method employs higher nositive-negative sample filtering threshold compared to the ControINet-SPO (0.8 v.s. 0.4). To visually validate the enhanced diversity of the candidate pool, Fig. b) specifically compares the maximum aesthetic score differences in the candidate pools be- tween the two methods across different denoising stages. Experimental results demonstrate that our method achieves significantly higher maximum aesthetic score differences than ControlNet-SPO at all denoising stages, directly confirming that the requence-guides sampling strategy effectively enhances diversity characteristies in candidate samples.
Mar senii mure offf
03-
Rutton
55200
52
Neam
Suat
Se
Sexar
Se-1
10
13 34 15
36
Timesdopo
Figure 5: The ablation study results, (a) presents the performance comparison of rollback mecha- nisms guided by sketches of varying abstraction levels under different evaluation metrics. (b) shows the statistical analysis of score differences between positive and negative samples in the candidate pool.
The Impact of Different Abstraction Levels of Sketch in SGR Mechanism.
In Fig- 5ia), we compare the performance of rollback mechanisms guided by sketches at different abstraction levels through
evaluation. For visualization convenience, the values of HPS v2 (Wu et al. 2023). PickScore (Kirstain et al., 2023). and CLIP-Score (Hessel et al. 2021 are scaled down by factor of 30. Six comparative schemes are designed: SGPO denotes our method without the rollback mechanism; $1 represents the rollback strategy guided by the first sketch stroke; 50.2N; $0.4N, S0.6N, 80.8N correspond to rollback strategies guided by sketches at 20%-80% completion stages; S denotes the strategy guided by the second-to-last sketch stroke. Experimental results reveal significant performance differences across schemes: the SN-1 g--idddd strategy performs worst, while the $ su-ddd mechanism surpasses others in comprehensive evalua- tion. This phenomenon aligns with the theoretical derivation in Eq. S as the discrepancy between the text condition and sketch condition c,, increases, the cumulative information gain can be aug-
8

---

Published as a conference paper at ICLR 2026
$1.22
Ro.4W
Sp--1
Mushroom growing
Lonely
Skyscraper,
looking Xmaz frez ornaments,
Figure 6: Visualization esults to the impact of different abstraction levels of sketch in SGR mecha- nism.
mented, thereby effectively amplifying both the semantic-structtt information of conditions and the preference information of the model. Visual demonstrations are presented in Fig. 6
As shown in the Fig. 6 a closer examination reveals that the level of sketch abstraction used for rollback guidance has impact the control over image details. When guided by the more abstract sketch 81, the generated results maintain stable quality while better preserving the structural features of the sketch. However, as the number of strokes increases, this consistency gradually declines.
5.5 ON-THE-B
SKETCH-TO-IMANE IMAGE GENERATION TEST
During the dynamic process of sketch drawing, advanced sketch-to tettc-to--image generation model must demonstrate the capability to produce high-quality images from sketches at any drawing stage. To evaluate this, propose the On-the-fly Sketch-to-
Generation Test Leveraging MasaCtrl's (Cao et al., (023) foreground editing strategy, we effectively stabilize the generated background to eliminate interference. As shown in Fig. 7 our method generalizes well: even with highly abstract sketches or only few strokes, the generated images align with human preferences and accurately reflect input conditions. As sketches gain detail, the outputs evolve accordingly, demonstrating the model's 's practical value in interactive creation.
WENTE
noied up edge
in pence pareet parch
Upright guhoo
Figure 7: Our method generates natural, conditionally accurate images from sketches of varying abstraction, with results increasingly matching user expectations drawing progresses.
CONCLUSION
In this paper, we propose a novel framework named SketchEvo, which leverages drawing dynamics to enhance image generation. Specifically, during the training phase, introduce sequence- guided optimization strategy to enhance the diversity of candidate samples, thereby improving the alignment of generated results with human preference. In the inference phase, sequence-guided

---

Published as a conference paper at ICLR 2026
rollback mechanism is adopted: the initial sketch strokes used to guide rollback, ensuring that the generation process aligms with human preferences while meeting user conditions. Experimental results show that our method effectively breaks through the trade-off bottleneck between creativity and structural fidelity in traditional models.
ACKNOWLEDGMENTS
This work was supported in part by the Postdoctoral Fellowship Program of the China Postdoctoral Science Foundation (CPSF) under Grant No. GZC20251088.
ETHICS STATEMENT
This research strictly adheres to the ICLR Code of Ethics. All used datasets are publicly available and comply with the relevant licenses and privacy policies. The study does not involve human subjects or sensitive information, and contains no discrimination, bias, or harmful impact. There are no conflicts interest.
REPRODUCIBILITY STATEMENT
We place great importance on the reproducibility of this research. Detailed descriptions of the model architecture and key experimental settings are provided in both the main text and the appendix. Anonymized and downloadable source code has been submitted as supplementary material, and www.seeec.cseceeee/seeeeeeeeeeee
can be accessed at: steps for the datasets used in the experiments, as well as the complete experimental procedures, are thoroughly presented in the supplementary materials. Theoretical assumptions and detailed proof' of the main conclusions are included in the appendix.
The preprocessing
REFERENCES
Quick, draw! htttp::/ggu.ccdraaw...ttoooooo/
2016.
Kevin Black, Michael Janner, Yilun Du Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In ICLR, 2024.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023.
Yu Cao, Xiangqiao Meng, PY Mok, Tong- Yee Lee, Xueting Liu, and Ping Li. Animediffusion: Anime diffusion colorization. IEEE Transactions Visualization and Computer Graphics, 30 (10):6956-6963 2024.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In WACV, 2024.
Zhipeng Chen, Lan Yang, Yonggang Qi, Honggang Zhang, Kaiyue Pang, Ke Li, and Yi-Zhe Song. Versagen: Unleashing versatile visual control for text-to-image synthesis. AAAI, 2025.
Pinaki Nath Chowdhury, Aneeshan Sain, Ayan Kumar Bhunia, Tao Xiang, Yulia Gryaditskaya, and Yi-Zhe Song. Fs-coco: Towards understanding of freehand sketches of common objects in con- text. In European conference on computer vision, pp. 253-270 Springer, 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In JCML, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. In NewrlPS, 2023.
10

---

Published as a conference paper at ICLR 2026
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image dif- fusion models without specific tuning. In The Twelfth International Conference Learning Representations, 2024.
Qingdong He, Jinlong Peng, Pengcheng Xu, Boyuan Jiang, Xiaobin Hu, Donghao Luo, Yong Liu, Yabiao Wang, Chengjie Wang, Xiangtai Li, and Jiangning Zhang. Dynamiccontrol: Adaptive condition selection for improved text-to-image generation. CoRR, 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-fiee frrennceefreeeeeee evaluation metric for image captioning. In EMNLP, 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Jijin Hu, Ke Li, Yonggang Qi, and Yi-Zhe Song. Scale-adaptive diffusion model for complex sketch synthesis. In ICLR, 2024.
Minghui Hu, Jianbin Zheng. Daqing Liu, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, and Tat- Jen Cham. Cocktail: Mixing multi-modality control for text-conditional image generation. In NeurIPS, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Wan- Yen Lo, et al. Segment anything. In ICCV, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, pp. 36652-36663, 2023.
Subhadeep Koley, Ayan Kumar Bhunia, Deeptanshu Sekhri, Anceshan Sain, Pinaki Nath Chowd- hury, Tao Xiang, and Yi-Zhe Song. It's all about your sketch: Democratising sketch control in diffusion models. CVPR, pp. 7204-7214, 2024.
Black Forest Labs. Flux: Official inference repository for flux.1 models. https:////thhb. flbbs/flux.. git 2024. Accessed: 2024-11-12.
Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback: Project page: liming-ai. github. io/controlnet.plee ppus.ppuuppppp ppuus.ppu.... ECCV, 2024.
Zhanhao Liang Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization. 2025. URL htt.:/tsss//aaai.oor////a/a////e
http://aaxvvv.. http://bs//2440 04314
Bai LiChen, Shitong Shao, zikai zhou, Zipeng Qi, zhiqiang xu, Haoyi Xiong, and Zeke Xie. Zigzag diffiusion sampling: Diffusion models can self-improve via self-reflection, 2025.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Xiaoyu Liu, Yuxiang Wei, Ming Liu, Xianhui Lin, Peiran Ren, Xuansong Xie, and Wangmeng Zuo. Smartcontrol: Enhancing controlnet for handling rough visual conditions. In ECCV, 2024.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023.
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, 2024.
11

---

Published as a conference paper at ICLR 2026
Pouyan Navard, Amin Karimi Monsefi, Mengxi Zhou, Wei-Lun Chao, Alper Yilmaz, and Rajiv Ramnath. Knobgen: Controlling the sophistication of artwork in sketch- based diffusion models. arXiv preprint arXiv:2410.01595,02 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, pp. 27730-27744, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. 2024.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang. Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, and Ran Xu. Unicontrol: A unified diffusion model for controllable visual generation the wild. CoRR, abs/2305. 2023a.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, and Ran Xu. Unicontrol: A unified diffusion model for controllable visual generation in the wild. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023b.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, pp. 13728-537444 2023.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-imaat -mage diffusion models for subject-driven generation. In CVPR, 2023.
Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions Graphics (TOG), pp. 1-12, 2016.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In CVPR, 2024.
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancedif- fusion: Instance-level control for image generation. In CVPR, 2024.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis, 2023.
Jiazheng Xu, Xiao Liu, Yuchen WU, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, pp. 15903-15935, 2023.
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin. Shurun Li, et al. Visionreward: Fine-grainer ence learning for image and video generation. arXiv preprint arXiv:2412.21055 2024.
dimeniiaimnniinnal human prefer-
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In CVPR, 2024.
12

---

Published as a conference paper at ICLR 2026
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0672,, 2023.
Po-Hung Yeh, Kuang-Huei Lee, and Jun-Cheng Chen. Training-free diffusion model alignment with
sampling demons. arXiv preprint arXiv:2410.0576, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In ICCV, 2023.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness deep features as perceptual metric. In CVPR, 2018.
Tao Zhang, Cheng Da, Kun Ding, Kun Jin, Yan Li, Tingting Gao, Di Zhang, Shiming Xiang, and Chunhong Pan. Diffusion model noise-aware latent reward model for step-level preference optimization. arXiv preprint arXiv:2502.0105. 2025.
Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. ECCV, 2024.
13

---

Published as a conference paper at ICLR 2026
Technical Appendices and Supplementary Material
A EXPERIMENTAL DETAILS
In this section, we provide additional details on the dataset, experimental setup, and evaluation metrics.
Datasets.
Sketchy (Sangkloy al. 2016 dataset. The Sketchy dataset comprises 125 object categories, with each category containing 100 real images and corresponding 7-10 hand-drawn sketches along with their drawing sequences, totaling 12,500 real images and 75471 associated sketches. To ensure ex- perimental rigor, we employ stratified sampling strategy to partition the data into non-overlapping random splits at an 8:2 ratio.
FSCOCO (Chowdhury et al. 2022) dataset contains 10,000 scene sketches drawn by 100 non- professionals, each with stroke-level temporal data and text descriptions. Collected under memory- based drawing setup, it captures both complex structures and human subjective preferences, making well-suited for evaluating models in multi-object scene generation.
QuickDraw (qui 2 is a large- scale sketch dataset from Google's "Quick, Draw!" game, contain- ing 50 million sketches across 345 categories drawn by millions non-expert users. Constrained to 20 seconds, the sketches are highly abstract, often with distortions or missing parts, creating clear gap from real images. In this work, we use a 2.5K subset to evaluate our model's ability to generate images under such abstract conditions, focusing on its alignment with human preferences and its capacity to interpret and repair incomplete structures.
AnimeDiffusion (Cao al. 2024) dataset is benchmark specifically designed for anime face line- art colorization, created through face cropping, aligmment, and denoising. It consists of 31,696 training samples and 579 test samples, with resolution of 256x256. Each colored face image is paired with a corresponding line drawing generated using the XDoG algorithm, featuring diverse line styles and a high level of professionalism, making well-suited for evaluating model performance under professional sketch conditions.
Settings.
During training, we employ the Adam optimizer with a learning rate of 1 x 10-5, a batch size of 4, and perform 4 training epochs. The diffusion model is configured with a total of 20 timesteps, and the candidate pool size K is set to 5. The initial sketch 81 and the final complete sketch sn are always selected, while 3 additional sketches are chosen from the remaining sequence as conditions to guide the denoising process for generating candidate pool samples. A threshold of 0.8 is applied for filtering positive-negativy sample pairs, with timesteps 10 20 designated as the late denoising phase.
Metries.
PickScore (Kirstain et al. 2023) developed Pick-a-Pic, large open dataset consisting of textto- image prompts and real user preferences for generated images. They then utilized this dataset to train CLIP-based scoring function, PickScore, for the task of predicting human preferences.
ImageReward (Xu et al. 2023) developed ImageReward, the first general-purposs text-to-image human preference reward model, which is trained based on systematic annotation pipeline, including rating and ranking and has collected 137,000 expert comparisons to date.
HPS v2 (Wu al.) 2023) first introduced the Human Preference Dataset v2 (HPD v2), a largescale dataset comprising 798,090 human preference choices on 433,760 760 pairs of images. By finetuning CLIP using HPD v2, they developed the Human Preference Score v2 (HPS v2), scoring model that more accurately predicts human preferences for generated images.
Sketch-LPIPS (Zhang al. 2018) is used to evaluate the perceptual similarity between generated image and a sketch. extracts the target region from the generated image using SAM (Kirillov al. 2023), computes its Canny edge map, and compares it with the original sketch. Then, features
14

---

Published as a conference paper at ICLR 2026
extracted using deep network (e.g., VGG) to calculate the LPIPS score, where lower score indicates higher similarity.
CLIP-Score (Hessel et al. 2021 metric used to measure the semantic consistency between an image and its text description. uses the CLIP model to extract features from both the image and the text, then computes the cosine similarity between them. A higher score indicates better alignment between the image and the text.
B ABLATION
PS San
(a)
(b)
Figure 8: Ablation study results: (a) illustrates the effect of candidate pool size on generation quality under the SGPO mechanism, while (b) shows the impact of different scoring models on the experi- mental outcomes.
Ablations on how SPO performance change w.r.f. sample diversity.
As we can see Fig. (a). when l: increases from 5 to 7, the marginal performance gain is substan- tially lower than that observed between = and = 5, revealing an inherent optimization bottle- neck. The underlying mechanism is attributed to the progressive homogenization of sketch states: as more intermediate sketches are sampled, the state differential between 8n and Sn-t decays, driving performance improvements toward asymptotic plateau. Increasing k also incurs significantly higher computational cost for candidate pool construction, reducing training efficiency. In summary, considering both sampling efficiency and generation quality, adopt k the default setting.
The impact of scoring model.
To evaluate the sensitivity of our method to the choice of scoring model, we conducted additional experiment using PickScore as the scoring model. The results of this experiment are presented the Fig. (b). Our method shows strong robustness to the choice of scoring model, evidenced by the small performance differences across most metrics (HPS v2, PickScore, IPPPS, and CL.PP-ccore) when evaluated with different scoring models.
C PSEUDOCODE FOR THE INFERENCE STAGE IN THE SGR MECHANISM
Algorithm Inference Stage of SGR Mechanism
Require: Text prompt Complete sketch Can- Guidance sketch Ca,, Inference steps T Ensure: Clean image Xi
1: FT ~ ~N(00
2: for = T down to 1 do
3:
It-1 t T6 (#x
C, C
4: 2t - [x
C, Co t)
5: Iz-1 $ (Tx Cogg
6: end for
7: return 20
15

---

Published as a conference paper at ICLR 2026
D PROOFS
In this section, we provide a mathematical derivation to validate Eq
Diffusion Model. We define T the total number of denoising steps, c the conditonal prompt, Cep, as the n-th sketch sequence. The denoising process can be defined as 4 : :W x C - D, where N is the Gaussion distribution and D is the target data distribution. Given a starting point 2T N, D, where is the condition guidance scale during denoing. The mapping corresponds + to the probability P(role, Cs,, 71, For simplicity, we only consider the initial input in Φ. Similarly, we can reverse the process using an inversion function D x C + N. Under a inversion guidance scale 72. we obtain inverted data ëт = ve (Tolc, C2 72) N from žo E D. Following DDPM (Ho et al. 2020),, we treat diffusion model as a Monte Carlo process and decompose into times single-step denoising mappings as
we can generate 2n ((Tric, Ca., 7)
(Err|c, Can. x" (xx (erlec Csn: (esm:))>> o ...... D² (2222 Csn. ())::))>>>))))))))))>))) D (2111c, Con. 71).
(9)
And define "²
2t-1 = (2' Co.. ?1): = vai-1 Dt VI- (ic)(z))
(10)
Var
where Or = II (1 B.) and B, are the pre-defined parameters for scheduling the scales of
adding noises in DDIM (Song al. 2021a) scheduler. We denote eg (xxg) the predicted score by the denoising network 0 at timestep
Similarly, we obtain ēz-1
(Z+ 1/c, CSn 72)
an
84-1 + Voi
c (T+-))
(1I)
V
a1-1
In Eq. 11 we approximate the score predicted at timestep with timestep / - along the inversion path, set e (# Ra e.(ū,) When the approximation error is negligible, & and V can be proven
to be inverse functions (Mokady et al. 2023). meaning that T = Φ- Therefore, we can represent Te
Zo V - (ir (c Co.. 72) ((ārlc, : 72).
(12)
Classifier free guidance. In the classifier-free guidance approach (Ho & Salimans 2021). the score prediction model ug is trained rained both conditionally and unconditionally. At inference time, the denoising
is obtained by blending the conditional and unconditional outputs of uo ,allowing flexible control over the strength of guidance through a tunable scale. Specifically, for denoising and inversion process, use guidance scales 7i and 72, with the corresponding scores as Eq. 7 where We is the noise predictor, and 0 is the null prompt, representing the denoising result under unconditional settings.
Derivation. Given inference timestep of T, we can obtain the inverted latent Z7 as
JT
Var
(Var
(Var-1
o (ĀT-1)-
(13)
Qr-1
oT
0T-1
For the sake of convenience, set
MT
VT VaT
(14)
OT-1
OT
OT
Through iterative and combinatorial processes, Ty could be expressed as İT mrār-1 + n (i)-1)
T 1 + MTTTT 10 (āT (āT-2)
+ NTTg (āT-1)
MPT0r- + MTMT- - T-2 (āT + mTMT - nnee ((T-1)
m Σ T4 II m
(18)
k=1+1
(15)
16

---

Published as a conference paper at ICLR 2026
Similarly, perform iterative derivations to obtain the equivalent form of Fy
Jy II
m,Jo
The II
mx))(a))-
(16)
k=t+1
Then in Z-sampling, we focus solely on local cycle of æt - 21-1 - Zt. Substituting Eq. into Eq. yiell 2, as
an-1)at (g((0))
Tz St v1 arcb((0))
04-1
1
var
(V Ott
- 1
E (Tx-1)
a--1
(1-aa-1e (1-ac-1)00 (cg(x)) <)((t-1)) (17)
2t + v -ct (+b(d)=1)
<t()))) + #
04-1
1-
2x +
VT- -
(eb(zr) - cb(()-1))
0x-1
Tt + vo
(Vo.
-
0tt-1 -1)
(cb(
- (t)())1)
We define the latent difference of Z-Sampling accumulated
T
òz-Sampling
(ot 2z)²
T ach? (c)(x)) - ())
(18)
t=1
2
T
Σ
anli
-
(Gx) - (Tx-1)
Ty Eesemnnticc information gain term
rpproximmmiioonn eror term
Excluding the approximation error introduced by inversion algorithm, we can rewrite Eq.
18
8z - Sampling
Σ andit (cg(are) - c).)))
(19)
Thus, we have demonstrated that Öz-Sampling proach results in H, and & being the same at each timestep t, from Eq. 7 we note that (zz) and (ā,) are obtained under guidance scales 7 and 72 respectively. Thus the effect of Z-sampling is further equivalent as
Although the step-by-step ap-
T
8z Sampling 8z-Sampling
Σ
achi + (x Cav, t) u1u 0, Cav: t)
(20)
(1 + 72)u2 (Tz, c, c 1) + u2 (T 0, Co., 1))
In our experiments, we adopted the parameter settings from ZigZag, setting 72 0, which simplifies the above equation as follows:
8z Sampanggg
Σ an ((1 (1 () 71 71u0(re 0, Cox,
uo (Te e, Co., 1))
(21)
At a given time step t, since Sy remains constant, thus, Sampling & (uo C, Cza t))² At this point, the proof of Eq. 8 has been completed.
17

---

Published as a conference paper at ICLR 2026
E VALIDATING THE MODEL'S TRANSFERABILITY TO EDGE-MAP CONDITIONING
To validate the transferability of our model, we applied our model to edge maps. Specifically, we first convert the corresponding images in the COCO (Lin al., 2014) dataset into edge maps, then transform them into SVGs to obtain separable contour segments. From these segments, we extract pseudo "initial sketch" 51 for the SGR mechanism. This process does not rely on real stroke order, nor does it require users to provide the drawing process.
As show in Tab. and Fig. 9 the experimental results show that, the performance in the pseudo- sequence mode remains largely comparable. The generated outputs remain stable in terms of struc- tural consistency and aesthetics. These findings demonstrate that our pre-trained model: (1) can operate effectively even with only the final sketch; (2) maintains stable inference performance with- out stroke sequence information.
Table 3:
Performance comparison demonstrating the model's robustness and adaptability when trained on sketches and applied to edge-map inputs.
Method
ImageReward
HPS v2
PickScore
LPIPS-Sketch
CLIP-Score
ControlNet
0.023
24.85
20.02
0.43
26.01
SOO
0.77
28.05
22.16
0.48
26.23
Ours
0.96
30.53
22.71
0,46
26.63
In summary, our method demonstrates excellent adaptability and robustness in practical scenarios, maintaining high-quality generation performance even without real drawing sequences.
Cortroliin
Shetch
Nentrolie
Controlliet-
There coveral stufted onimals Tegether o bed
Applc
Seven OurTccies are stocked Top o
Wasting
toble
food
that
fird of
Left: images generated under edge-map guidance. Right: corresponding example images used in the human study.
Figure 9:
Table 4: Performance comparison of methods across aesthetic, structural, textual alignment, and
realism metrics.
Method
Aesthetics Structure Match Text Alignment Realism Naturalness overall
ControlNet
3.60
4.68
4.71
4.08
0.8%
SontrolNet-SPO
4.38
4.28
4.79
4.22
16.53%
ours
4.88
4.56
4.95
4.76
82.67 %
F HUMAN STUDY
We conducted human study to evaluate objective quality against real user preferences, comparing Ours with ControlNet and ControlNet-SPO. Twenty five "sketch + text" pairs were used, with each model generating corresponding images presented anonymously in random order. Participants rated Aesthetics, Structure Match, Text Alignment, and Realism on a 5-ionnt Likert scale, and provided an overall preference (A/B choice). Fig. D shows example images from the study. Tab. 4 indicate
18

---

Published as a conference paper at ICLR 2026
that method is consistently preferred across metrics, confirming its superior objective quality and alignment with user preferences.
G MORE SAMPLE IMAGES GENERATED BY SKETCHEVO Giraffe
Chair
Jellyfish swimming
Butterfly
Rusted sedan
Blimp
Pineapple
Hot Dog side
Bread
Zebra
Helicopter
Harp
Figure 10: More visualization results for the Sketchy datasets.
Ambulance
Bush
Bicycle
Barn
6
Bed
Shoe
Bench
Cell phone
Bulldozer
Helme
Couch
Laptop
Figure 11: More visualization results for the QuickDraw datasets.
Hand style portnoit.
drawn onine style portroit
drawn style portroit.
Hand style portroit.
drown style portroit.
onime style portroit.
drown style portroit.
style portnot.
Hond style drown orime style portroit.
Hond drown onime style partroit.
drown onime style portroit,
Figure 12: More visualization results for the AnimeDiffusion datasets.
19

---

Published as a conference paper at ICLR 2026
A statue of sitting
Two motor bikes parked in front other.
Acrapaan is waiting.
Sheep grazing on the hill.
SM
garden with flowering plants and birds.
Two giraffes standing near
train on track.
plane flying in sky.
A
M1
rancher gathering up his cows in farm.
sitting elephont.
A church ond house witha by its
A boot ocean.
Figure 13: More visualization results for the FSCOCO datasets.
Figure 14: More visualization results off- prompt on the QuickDraw dataset.
H VISUALIZATION RESULTS OF OFF-PROMPT ON THE QUICKDRAW DATASET
We conduct inference using only the QuickDraw sketches as input. The conclusions are as follows:
Clear-contour sketches yield consistent structure-driven results. For sketches with clear and characteristic contours (e-g., lighthouse, cello), the model can still produce semantically aligned images that fit natural scenes even without textual input, suggesting that well-defined structural cues alone can anchor the intended concept.
Semantic ambiguity in minimalist sketches. For highly abstract or extremely sparse sketches, removing text introduces substantial semantic uncertainty. For instance, when the sketch merely circle, sampling with different random seeds produces diverse and often mutually inconsistent outputs. This demonstrates that textual input is essential for stabilizing and disambiguating the semantics of minimalist drawings.
20

---

Published as a conference paper at ICLR 2026
Complementary control beyond structure. In addition to semanties, text typically govems appearance-related attributes such as color, number, material, background composition, lighting, and stylistic preferences facttennce--fctorss not specified by the sketch.
Taken together, these observations highlight the complementary roles of the two modalities: the sketch primarily constrains the structural outline, while the text refines high-level semantics and controls appearance attributes, jointly shaping the final generation.
I THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this work, we employed a large language model to assist in polishing the writing and improving the clarity and readability of the text.
21


Reviews:
[{"review_id": "omvEXEBXEQ", "reviewer": "Reviewer_TQFU", "summary": "This paper proposes the SketchEvo framework, which harnesses the dynamic evolution of sketches for enhanced image synthesis. The key insight is that intermediate sketches from different drawing stages represent varying levels of abstraction and detail. These sketches offer meaningful semantic and structural divergence while maintaining a connection to the user's intent.", "questions": "- Why using intermediate sketches from different drawing stages enhances human preference for generated images?\n- Does the performance gain come from using intermediate sketches or simply from increasing the number of samples?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a novel framework that leverages the sketch sequences for preference-based optimization.\n- The experimental results demonstrate improvements over baseline methods.", "weaknesses": "- The paper does not clearly justify why using intermediate sketches from different drawing stages enhances human preference for generated images. While intermediate sketches offer more structural divergence and variation, it remains unclear how this divergence and variation improve human aesthetic preference. The early sketch strokes actually contains very little information of the full structure.\n- It would be interesting to see the results if the sampling sketch condition $c^k_{s_n}$ were replaced with a constant value $c^k_{s_N}$ while sampling different noise z to ensure sample diversity. To my understanding, this sampling method differs from ControlNet-SPO. This experiment would verify whether the performance gain comes from using intermediate sketches rather than simply increasing the number of samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ISzrUstv7c", "reviewer": "Reviewer_tnmo", "summary": "SketchEvo seeks to improve sketch-to-image generation by analyzing the drawing process rather than treating sketches as static blueprints. SketchEvo captures the evolutionary stages of drawing—from initial rough strokes to the completed sketch—to understand what the artist prioritizes and intends. The claim is that by recognizing that the sequence and refinement of strokes reveal important information about the drawing's intended meaning, SketchEvo generates images that better align with human preferences and aesthetic sensibilities.\n\nThe authors state that the system achieves these improvements through two innovations: a training method that uses sketches at various completion stages rather than random variations to learn aesthetic preferences more effectively, and a generation process that employs initial sketch strokes to guide a \"rollback\" mechanism balancing structural fidelity with natural beauty. \n\nThe results are stated as demonstrating superior aesthetics with higher human preference scores.", "questions": "It wasn't clear whether or how the model and the representations captures the artist's intent in an explicit manner. An explanation of that would be helpful.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The notion of capturing artist **intent** in the generation and refinement of an image is a novel idea that this reviewer appreciated. This is a well-written paper. The techniques employed seemed sound. Results were interesting. Ablation studies were good.", "weaknesses": "What seemed to be missing is an explicit capturing of the artist's intent with a view towards having the model provide explanations and also learn high level concepts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aXnC8AYwzN", "reviewer": "Reviewer_1prC", "summary": "The paper introduces SketchEvo, a sketch-guided image generation framework that leverages the drawing sequence—from early strokes to the completed sketch—rather than treating the final sketch as a static constraint. The core claim is that existing text+sketch methods underperform because they ignore preference signals implicit in the sketching process and struggle to produce sufficiently diverse positive/negative samples for preference alignment under multimodal constraints. \n\nSketchEvo tackles this with two components: \n\n* Sequence-Guided Preference Optimization, which uses sketches at different completion stages to create meaningfully divergent training pairs and yield stronger gradients for preference learning; \n\n* Sequence-Guided Rollback, an inference-time mechanism that balances textual semantics with sketch structure by integrating the sketch sequence into rollback updates so that learned aesthetic preferences transfer to generation while preserving structural fidelity.", "questions": "* **Metric validity for structure:** Sketch-LPIPS can be biased by edge density. Can you add a complementary metric (e.g., edge chamfer distance, keypoint F-score) and/or a small human “structure-match” study?\n\n* **Training data specifics:** How many unique stroke sequences per class were used? Any filtering for noisy/misaligned strokes? Please clarify sequence quality controls and their impact.\n\n* **Robustness to sketch noise:** How sensitive are results to (a) shuffled stroke order, (b) dropped early/late strokes, (c) jittered stroke geometry, and (d) off-prompt doodles? \n\n\n**I am open to changing my score based on the author's responses.**", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* Clever use of the sketching sequence (early→late strokes) to create structured candidate diversity for preference learning; pairing SGPO with SGR is a fresh, problem-driven combo.\n\n* Method is well-specified; ablations on candidate-pool diversity and rollback settings support the mechanism(however, they didn't mention the number of GPUs that they used for training and just mentioned the type of GPU); consistent wins on preference and sketch-fidelity metrics across multiple datasets.", "weaknesses": "* **Lack of related work & baselines for hand-drawn sketches.** The paper mostly compares to ControlNet-style direct sketch conditioning. It omits stronger, recent sketch-specific approaches such as Democratising Sketch Control [2] and KnobGen [1], which explicitly address amateur/sparse sketches—neither discussed nor used as baselines. Add these to Related Work and include at least one competitive replication.\n\n* **Bad Experiments, need more robust results**: As the author shows in Figure 6, s_i affects the generation. ControlNet and the T2I-Adapter have coefficients that the client can control to receive a different level of condition on spatial information. Did the author change the coefficient for those baselines?\n\n* **Proxy metrics only for aesthetics.** Claims rely on ImageReward/HPSv2/PickScore; no human A/B tests or significance. \n\n* **Requires stroke sequences.** Many users provide only the final sketch. Assess a “no-sequence” mode (pseudo-sequences via edge reveals/stroke segmentation) and report the performance drop.\n\n* **Cost underreported**. SGPO candidate pools and SGR rollbacks add compute. Report wall-clock per step, GPU hours, VRAM, and quality–cost curves vs. K and rollback steps.\n\n* **Effect of text on generation**: I think text has a lot of effect on the generation. We can see this effect in Figure 10. What is the effect of text on generation? Do they have any experiments without text and only rely on the sketch?\n\n\n\n\n\n\n\n[1] Navard, Pouyan, et al. \"KnobGen: controlling the sophistication of artwork in sketch-based diffusion models.\" arXiv preprint arXiv:2410.01595 (2024).\n\n[2] Koley, Subhadeep, et al. \"It's All About Your Sketch: Democratising Sketch Control in Diffusion Models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vHwzTCbFrm", "reviewer": "Reviewer_bvYj", "summary": "The paper proposes SketchEvo, with two parts:\n- SGPO: extend SPO-style pairwise preference optimization to sketch-guided diffusion by building the candidate pool with different sketch stages (not just noise like the prior works) and picking top/bottom by a pretrained preference scorer.\n\n- SGR: an inference-time sequence-guided rollback that blends conditional/unconditional scores with text+sketch (two γ’s), arguing increased “information gain” when text and sketch diverge.\n\nevaluated on Sketchy (train), and tested also on QuickDraw, AnimeDiffusion, FSCOCO. metrics: ImageReward / HPSv2 / PickScore (aesthetics), CLIPScore (text alignment), and a sketch-LPIPS variant (sketch fidelity). baselines include ControlNet, T2I-Adapter, VersaGen, and ControlNet with DPO/SPO.", "questions": "On reward hacking and evaluation metrics: My primary concern is the potential for \"reward hacking\" or circularity in the evaluation. The paper states that SGPO uses a \"pretrained scoring model\" (L195) to select the winning $x^w_t$ and losing $x^l_t$ samples during training. The evaluation in Table 1 then relies on automated preference metrics like ImageReward, HPSv2, and PickScore.\n\n- Could you please clarify which specific \"pretrained scoring model\" was used for training?\n- If this training scorer is one of the evaluation metrics (or from the same family, e.g., another CLIP-based preference model), how can we be sure the model hasn't simply learned to optimize for the biases of this specific scorer, rather than for genuine, generalizable human aesthetic preference?\n\n\nNecessity of human evaluation: Following the previous point, the gains on automated preference scores are strong, but their correlation to actual human judgment is not guaranteed, especially in a dual-condition (sketch+text) setting.\n - Given the risk of reward hacking, a human evaluation study (e.g., pairwise preference tests comparing \"Ours\" vs. \"ControlNet-SPO\" and \"ControlNet\") seems crucial to validate the paper's central claims. Would it be possible for the authors to provide such a study, even on a smaller scale, to confirm that the observed metric improvements translate to tangible, human-perceived quality gains?\n\n\nMissing SOTA baseline comparisons: The related work section and introduction cite several strong, recent methods for controllable generation, such as SmartControl  and ControlNet++, which are highly relevant.\n\n- Could the authors provide a quantitative or qualitative comparison against these (or similarly strong) recent baselines? If this is not feasible, could you please provide a more detailed discussion on how SketchEvo's contributions differ from and are expected to perform against these methods, which also aim to improve fidelity in controllable generation?\n\n\nSGPO Mechanism and Learning Signal: In the SGPO formulation (Eq. 4), the model is trained on pairs ($x^w_t$, $x^l_t$) generated from different sketch conditions ($c_{s_w}$,  $c_{s_l}$)\nHow does this process ensure the model is learning a general \"aesthetic preference\" rather than simply learning to \"prefer more complete sketches\"? For instance, if $x^w_t$ is frequently generated from a late-stage sketch ($s_N$ ) and  $x^l_t$ from an early-stage sketch ($s_1$), is it possible the model is just learning to assign a higher likelihood to more detailed conditions?\n\n\nJustification for SGR using $s_1$: The ablation (Fig. 5a) suggest that using the most abstract sketch ($s_1$) for the rollback mechanism (SGR) is optimal, attributing this to maximized \"information gain\" from text-sketch divergence.\n- This is an interesting finding, but the theoretical justification in Appendix D relies on several approximations. Could you provide more intuition on why maximal divergence (using $s_1$) is superior to using a more structurally-informative intermediate sketch (e.g., $s_{0.6N}$ )? One might expect a more complete sketch to provide a better structural prior during the rollback.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple, intuitive idea: use sequence of sketches as a structured source of diversity ->  stronger positive/negative pairs -> better gradients than noise-only SPO. this feels original within sketch-conditioned alignment.\n\n- Method framing: clear diffusion-probability ratio objective (SPO/DPO family) and a roll-back mechanism adapted to dual conditions (text+sketch).\n\n- Decent empirical lift: consistent gains on preference metrics, with some qualitative improvements in composition/position/color, and cross-dataset demos.\n\n- Ablations: show candidate-pool size effects and performance vs. which sketch stage guides rollback; also a tiny “scoring model” swap test.\n\n- SGPO vs prior alignment: prior DPO/SPO/LPO/D3PO target text-only or step-aware preferences but don’t, to best of my knowledge, explicitly use sketch-sequence stages to create candidate diversity at each diffusion step. that is a novel and plausible extension for multimodal control.", "weaknesses": "- SGR vs rollback work: Rollback/zig-zag/self-reflection sampling has been explored for text-to-image; adapting it to joint text+sketch guidance with a sequence-aware blend is a reasonable incremental contribution. the theoretical “information gain increases with text–sketch divergence” is interesting, but depends on approximations and fixed γ’s.\n\n- Baselines: ControlNet, T2I-Adapter, VersaGen are appropriate; ControlNet-DPO/SPO adaptations are helpful. but given 2024–2025 progress, additional strong controls (e.g., ControlNet++ [1] /SmartControl [2] or more recent) would be useful; at least include one recent sketch-aware control baseline if feasible.\n\n- Possible circularity / reward hacking: The same family of preference scorers appears to be used both for pair selection during training and for evaluation, risking overfitting to those scorers rather than to human judgment. Cross-metric generalization is only partially demonstrated.\n\n- Limited human studies / no significance testing: No human pairwise evaluation targeted to sketch-conditioned synthesis. No confidence intervals, multiple seeds, or statistical tests are reported for the main tables.\n\n- Writing clarity: Background part (L126-149), is not clear and detailed enough. All the terms, should be explained clearly for readers to better grasp the discussion. For example, what is $x^w_t$, $x^l_t$ ?\n\n[1] https://arxiv.org/abs/2404.07987\n\n[2] https://arxiv.org/abs/2404.06451", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)