SketchingReality: From Freehand Scene Sketches to Photorealistic Images
Authors: Ahmed Bourouis
Mikhail Bessmeltsev
Yulia Gryaditskaya
Venue: ICLR 2026 Poster (2026)
Keywords: Image Generation
Freehand Sketches
Primary area: generative models

Abstract: Recent years have witnessed remarkable progress in generative AI, with natural language emerging as the most common conditioning input. As underlying models grow more powerful, researchers are exploring increasingly diverse conditioning signals -- such as depth maps, edge maps, camera parameters, and reference images -- to give users finer control over generation. Among different modalities, sketches constitute a natural and long-standing form of human communication, enabling rapid expression of visual concepts. Yet algorithms that effectively handle true freehand sketches -- with their inherent abstraction and distortions -- remain largely unexplored.
In this work, we distinguish between edge maps, often regarded as “sketches” in the literature, and genuine freehand sketches. We pursue the challenging goal of balancing photorealism with sketch adherence when generating images from freehand input. A key obstacle is the absence of ground-truth, pixel-aligned images: by their nature, freehand sketches do not have a single correct alignment. To address this, we propose a modulation-based approach that prioritizes semantic interpretation of the sketch over strict adherence to individual edge positions. We further introduce a novel loss that enables training on freehand sketches without requiring ground-truth pixel-aligned images.  We show that our method outperforms existing approaches in both semantic alignment with freehand sketch inputs and in the realism and overall quality of the generated images.

Published as a conference paper at ICLR 2026
SKETCHINGREALITY FROM FREEHAND SCENE SKETCHES TO PHOTOREALISTIC IMAGES
Ahmed Bourouis'
Mikhail Bessmeltsev
Yulia Gryaditskaya
1 University of Surrey, UK
Université de Montréal, Canada
3 AAdobe Research, UK
Reference Image used for skettch
Methods inputs Sketch
Generated images
ControINet SD2.1
ControlNet SDXL
Ours SD2.1
T21 SDXL
Horses in the garden area
bear in the forest
With just few strokes, sketches can convey complex visual concepts that are difficult to express in words, making them a natural conditioning choice for efficient, human-centered, control- lable generative AL. Our method takes imput freehand sketch together with text prompt. The figure compares our results with state-of-thh-ar baseline ControlNet (Zhang et al. 2023) and T21-Adapter (Mou et al. 2023) - freehand sketches from the FS-COCO dataset (Chowdhury et al. 2022). SD2.1 and SDXL. represent the used backbones: |Rombach et al. (2022) and Podeil al. (2023), respectively. The first column shows the reference image presented to participants, who recreated it from memory within a limited time, simulating how humans draw from a mental image. Our approach achieves a strong balance between sketch adherence and photorealism.
Figure I:
ABSTRACT
Recent years have witnessed remarkable progress in generative AI, with natural language emerging as the most common conditioning input. As underlying mod- els grow more powerful, researchers are exploring increasingly diverse condition- ing signals such as depth maps, edge maps, camera parameters, and reference images - to give users finer control over generation. Among different modal- ities, sketches are natural and long-standing form of human communication. enabling rapid expression of visual concepts. Previous literature has largely fo- cused on edge maps, often misnamed "sketches", yet algorithms that effectively handle true frechand sketches - with their inherent abstraction and distortions remain underexplored. We pursue the challenging goal of balancing photo- realism with sketch adherence when generating images from freehand input. A key obstacle is the absence of ground-truth, pixel-aligned images: by their ture, freehand sketches do not have a single correct alignment. To address this, we propose modulation-based approach that prioritizes semantic interpretation of the sketch over strict adherence to individual edge positions. We further in- troduce novel loss that enables training on freehand sketches without requiring ground-trutt pixel-aligned images. We show that our method outperforms exist- ing approaches both semantic alignment with freehand sketch inputs and in the realism and overall quality of the generated images. The code is available on our project webpage
For ControlNet and T2I-Adapter, we used parameter settings selected via user study and qualitative evaluation, detailed in Sec. 4.4.1 44..aand and Sec. C

---

Published as a conference paper at ICLR 2026
INTRODUCTION
In recent years, generative AI for visual content such as images and videos has progressed dramati- cally. While research continues to push the boundaries of quality and realism, enabling greater user control has become increasingly important. Researchers are exploring diverse conditioning signals complementary to natural language - such as depth maps, edge maps, camera parameters, and reference images. In this work, we focus human-drawn (freehand) sketches, which, alongside language, are among the oldest forms of human communication (Donald 1993). With just a few strokes, sketches can convey complex visual concepts that are difficult express in words, making them a natural conditioning choice for efficient, human-centered, controllable generative AI. Yet algorithms that effectively handle true freehand sketches with their inherent abstraction and dis- tortions - remain underexplored. Critically, we distinguish between edge maps, often regarded as "sketches" in the literature, and genuine frechand sketches. Perceptual studies show that freehand
sketches frequently represent abstracted object forms (eersky 2002 Eitz et al. 2012) and may
include distorted proportions or relative sizes (Hertzmann) (2225 Fig. I Iillustrates examples from
the FS-COCO dataset of scene sketches (Chowdhury et al., 22022. The first column shows the ref- erence image presented to the participants, who were then asked to recreate it from memory within a limited time, simulating how humans draw from a mental image. Objects in these sketches often exhibit high levels of abstraction - for example, in the second row of Fig. [ grass is represented by single strip of vertical strokes, while forest is depicted with a few schematically drawn trees. Relative sizes of objects also frequently deviate substantially from the reference images.
While freehand sketches provide rich semantic cues, their abstraction, distortions, and ambiguity make them challenging for existing generative models to interpret effectively. Existing condition- ing mechanisms often fail to capture the intended objects and relationships, producing images that either ignore important sketch details or compromise realism (Fig. []. These limitations highlight two key challenges for generative models: (i) extracting meaningful semantic representations from highly abstract sketches, and (ii) generating images that are both realistic and respect the sketch. In our work, we pursue the challenging goal of balancing photorealism with sketch adherence when generating images from freehand input.
First, we observe that for freehand sketches, the image quality of typical conditioning mechanisms, such as ControlNet (Zhang et al., (023) and Adapters (Mou et al. 2023), is limited by the latent features of their conditional imput encoders. Specifically, the VAE encoder used in ControlNet-basee models may lack semantic understanding of sketches, while the convolutional encoders in Adapter- based approaches may not be expressive enough to learn robust semantic representations. For the first time, we exploit semantic features from a fine-tuning its final layers for our task. To inject richer semantic features, we propose a dedicated modulation network (Fig. 2). This enables the encoder to maintain semantically meaningful latent space while capturing finer-grained visual details from sketches
sketch encoder (Bourouis et al.] 2024),
Second, we propose novel loss that sidesteps the need for pixel-aligned ground-truth images during training while preserving the semantics of the sketch input. To this end, we leverage the aforemen- tioned semantic sketch encoder estimate the likelihood of each sketch pixel belonging specific object category. During training, these semantic likelihoods guide the cross-attention between the corresponding textual tokens and the latent image representation, encouraging the generated image to follow the freehand sketch (Fig. 2p). We train on mix of synthetic sketches, algorithmically generated from reference images, and freehand sketches. This approach enables the preservation of semantic information, important details, and the generation of natural-looking images.
As sketches are often ambiguous for example, the trees in Fig. D (first row) are depicted as a set of ares - we rely on text as an additional input. Fig. D (Ours 1) demonstrates that our method strikes a strong balance between the realism of generated images and adherence to the sketch, preserving relative positions, orientations, and some finer details of objects, such as a bear facing left and the curve of its back (Fig. [ second row). We validate our designs via qualitative and quantitative comparisons with previous methods, well via perceptual user studies.
In summary, our contributions are: (i) new method that generates realistic images from freehand and potentially abstract scene sketches; (ii) the use of semantic sketch features through integration with the proposed modulation network in latent space; (iii) enabling efficient training on frechand sketches with loss function that emphasizes the semantic structure of sketch inputs.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Although earlier work on general sketch- or edge-based image generation relied on GAN-based approaches (Isola et al. 2017 Chen Hays, 2018: Huang et al., 2018: Liu et al. 2020 Wang al. 202I], recent work has demonstrated the superionity of diffusion- based approaches. In what follows, we focus our discussion on the latter. For a broader overview of spatial control in diffusion models, refer the reader to recent surveys (Cao et al. 2024 Jiang et al. 2024).
2.1 PIXEL ALIGNED
Earlier work enabling control via sketches often considers precise, pixel-aligned edge maps. Some methods incorporate sketch conditions by concatenating them along the channel dimension, either in the pixel space (Cheng et al. 2023) or in the feature space (Huang et al. 2023 Unlu et al. (2024). These approaches often produce high-quality, pixel-aligned results, but typically involve training or at least fine-tuning the entire model. To avoid the overhead of training full models, several methods introduce sketch guidance through lightweight stand-alone modules integrated into pre- trained diffusion models. ControlNet (Zhang et al.) (023) does this via a dedicated branch with zero convolutions, significantly reducing the number of trainable parameters. T21-Adapter (Mou al., 2023) trains lightweight convolutional encoder aligned with a U-Net denoiser. ControlNext (Peng al. 2024) uses a lightweight encoder learn conditional features and normalizes them match the distribution of denoising features, improving convergence. Our work is most closely related method proposed by Ham al. 2023), which modulates text- conditioned latents with sketches represented as three-channel RGB images and encoded by a UNet. Voynov et al. (2023) refine noisy latents using gradients of similarity loss between edge-guidance latents and the output of a trained MLP applied to multi-level U-Net features. However, we show that none of these methods generalizes reliably to freehand sketches, such as those Fig.
to the UNet-based encoder-decode
2.2 SEMANTIC GENERATION
Several works aim to improve sketch interpretation in generative models by focusing on the semantic content of sketch inputs. We categorize these approaches into two groups: methods that require training and those that are training-free.
2.2.1 TRAINED
Single-object DiffSketching (Wang et al.) 2023) targets single-object inputs and relies on classi- fier guidance and perceptual losses (LPIPS Zhang et al. (201 8a) and ResNet He et al. (0166). This strong reliance on classifier guidance makes it unsuitable for multi-object settings. CLAY (Zhang et al. 2024a) uses dense cross-attention to incorporate sketch conditions into 3D model generation, requiring a substantial number of training parameters. Koley et al. (024) replace text guidance with global single-object sketch encoding. To fine-tune the CLIP-based sketch encoder, they align the text and the sketch encoding during training and use a CLIP perceptual loss (Sain et al. 2023 Our work focuses scene sketches that usually contain multiple objects.
Multi-object Wu et al. 2023) fine-tune pixel-space diffusion model FS-COCO freehand sketches (Chowdhury et al. 2022). They train a ResNet-based sketch encoder with additional trans- former blocks from scratch, then pass those latent features instead of text denoiser. To boost realism on freehand sketches, a discriminator in an image space is trained. ZZhag et al. (2024b) pro- pose a multi-step approach that requires fine-tuning a diffusion model for each syntheynthetic scene sketch from SketchyCOCO (Gao et al.) 2020), where scenes are composed by assembling individ- object sketches. After manually segmenting objects, ControlNet generates their images, and unique identity embeddings are obtained using multi-concept inversion (Avrahami et al. 2023a). During inference, a blended latent diffusion model (Avrahami et al.] (023b) merges object and background representations at high noise levels, while later denoising steps are conditioned on back- ground prompts and object embeddings. However, depending on the noise schedule, outputs may appear unrealistic fail to reflect the intended object arrangement. Our method also targets free- hand scene sketches and is designed for latent diffusion models. It aims at photorealistic generation while closely adhering sketch guidance, without adding any inference-time overhead.
3

---

Published as a conference paper at ICLR 2026
Add
Modulation network
Encoder
noise
20(S,, Bi, 66, sx)
Soft
VAE
Bus waiting at the stop near tho clocktower
Loss supervision
Diffusion model
Text Encoder
St
Cosine
B₁
Similarity
Encoder
Sketch
Semantic
sketch features
Modulation approach
b. Attention supervision
Figure 2: Method overview. Modulation network (a). During training, the latent features 2, generated by applying a noising process to the clean latent 26, which is obtained by encoding the ground-truth image using the VAE encoder. During inference, 7 is instead sampled from a standard Gaussian distribution and then progressively denoised by the diffusion model. Text-conditioned diffusion model generates dime-dependent noise et, which we modulate relying on semantic sketch features, For details, please refer to Sec. 3.2 and Tab. A Attention supervision (b). Attention supervision allows us to train on combination of freehand sketches and sketches algorithmically generated from reference images. It bypasses the need for pixel-aligned ground-truth images which are unavailable for freehand sketches - and helps the modulation network focus on sketch semantics. For details, please refer to Sec. 3.4
2.2.2 TRAINING-FREE
Chen 2023) combines ControlNet with paint-with-wwe worhs guidance (Balaji al. 2022 which uses user-drawn masks to steer cross- cross-attention during inference to add semantic guidance. This approach sensitive to weighting, requiring careful tuning per category (Fig. TT]. Several works rely on the diffusion inversion of reference sketch (Mo et al. 2024 Ding et al. 2024). FreeControl (Mo et al. 2024) first generates coxticondt images to extract semantic and appearance bases, then transfers structural guidance by minimizing the distance between the basis coefficients of the gen- erated and reference images. Ding et al. (2024) update noisy latents using KI. divergence gradients between cross-attention maps from the current generation and those obtained by diffusion inversion of a reference sketch, showcasing the method only on single-object sketches. We also explore the properties of cross-attention maps, supervising them at training time with the introduced loss.
METHOD
In this section, we describe the key components of our method, their motivation, and the necessary background theory. provide additional implementation details in Sees. and
3.1 PRELIMINARIES ON LATENT DIFFUSION MODELS
Latent Diffusion Models (LDMs) improve the efficiency and scalability of diffusion models by op- erating in a compressed latent space instead pixel space. A pre-trained autoencoder first uses its encoder & to encode input images xn E I x into lower-dimensional latent representation 20 = Exo).
During training, Gaussian noise is gradually added to clean latent 2p over discrete timesteps (1...., T}. producing noisy latent at each step. A neural network with parameters 0 is then trained predict the noise eø (z,, t, c) added 2g each timestep optionally conditioned on some context (e.g., class label or text embedding). The model is trained using the following objective:
E Ezo.ett [|| co co(=,, (,)=)
Lnoise
where e ~ (V, I) is the Gaussian noise added at each timestep according to a predefined noise schedule.

---

Published as a conference paper at ICLR 2026
During inference, the model starts from Gaussian noise in the latent space and iteratively denoises using the learned reverse process. The final output is obtained by decoding the denoised latent 20(2, t) using the decoder D: 20 D(2)).
3.2 MODULATION NETWORK
To enable semantic image generation from abstract freehand scene sketches, we design a modula- tion network that refines the predicted noise c0(2,, Ctext) at each timestep, incorporating sketch guidance. Extending the ideas of noise modulation with spatial conditions HHam et al. (2023), we propose network explicitly designed exploit semantic sketch features. Our network re- ceives input the semantic sketch features, the latent 2,, and the predicted noise EΦ(.) and pro- duces scale S, and shift Be maps. These maps are used to modulate the noise prediction as detals
c2(2e,t t, Cterts Caketch) co @ (1 St) Bt. The next section Sec. 3.3 details how semantic sketch features are obtained. Here, we focus on the modulation network itself.
Our modulation network is encoder-decoder CNN architecture. As illustrated in Fig, 2 each input modality is first processed through separate downsampling path, projecting into an embed- ding space of equal dimensionality. The resulting feature maps concatenated and passed through a timestep-conditioned convolutional layer, followed by an activation function and another convolu- tional layer. The fused features are then processed by three upsampling blocks. The final output is split into scale S, E R #xx x4 and shift B, E IR xW x4 latent maps.
The model is trained with the loss defined in Eq. ( and additional regularization losses. Similarly to Ham et al. (2023), we apply L1 regularization on shift and scale parameters:
Cycale = ||Se|l1, Ly = |[
(2)
To promote more expressive and diverse transformations, we encourage variability in the predicted scale and shift maps by penalizing low variance across their values:
Lvar (a. + (B:)),
(3)
where a is the standard deviation computed over the elements of the modulation maps S₁ and B,.
3.3 SEMANTIC SKETCH FEATURES
To extract semantic features from freehand sketches, we leverage pre-trained CLIP-based encoder (Bourouis et al., 2024). This encoder was trained on freehand sketches from the FS-COCO dataset Choowdhur et al.J 2022) to perform open-vocabulary semantic segmentation of scene sketches. However, for the task of sketch-conditioned image generation, these features may lack sufficient fine-grained detail. Empirically, we found that fine-tuning the last few layers (three in our case) of the encoder significantly improves the alignment between the generated image and the input sketch.
3.4 TRAINING WITH FREEHAND SKETCHES
Training diffusion models with freehand sketches presents a key challenge: pixel-aligned, realistic reference images do not exist, and discussed in Sec. [ such images can rarely exist in practice. Using misaligned reference images, such as those in the FS-COCO dataset (Fig. [ introduces ambiguity in the standard denoising objective (Eq, (T).. To overcome this, we propose loss function that bypasses pixel-level correspondence and instead focuses on preserving the sketch's semantic structure.
Before introducing our loss, briefly review how text conditioning is commonly implemented in generative models via a cross-attention mechanism:
QK
Attention (2, K,V) = Softmax
MV,
(4)
Vd
where the query matrix Q is derived from the latent representation 2, while the key K and value V matrices are obtained from the text embedding The latent dimensionality is denoted by d. Each entry My in the attention matrix M captures the influence of the j-th text token on the i-th image patch, therefore defining in which spatial locations each particular semantic concept appears.
5

---

Published as a conference paper at ICLR 2026
We leverage the observation that the semantic encoder used in our modulation network, originally trained for segmentation, provides a strong spatial signal indicating the location and identity of ob- jects in the sketch (see Fig. 2p.). Therefore,
the features of the original pre-trained sketch en- coder (Bourouis et al. 2024) to compute ground-truth' attention maps Mynh- For synthetic sketches, rely the able segmentation maps in the MS-COCO dataset (Lin et al. 2014).
From the modulated noise cg(2,, f, Crezt: Caketch), we compute the denoised latent &0(=,, f, 26( (:)). We then pass Eo(-) through the denoising network (with text conditioning) to extract cross-attention maps M at multiple layers. We supervise these cross-attention maps M using ground-truth attention
maps Mgnt. following the loss formulation introduced by Sun et al. (2024) the context of layout- based image generation:
(1
(1-
Eprne M Ep
Lattnn 3EΣ 7ET JEZ 6,EB
EM M
Areg
Σ prub.
M pi
(5)
pi
here, M2 denotes the attention value at pixel for object P, in layer y. The set B contains spa- tial regions derived from Mgnh. and I is the set of valid text token indices. The loss encourages alignment of attention with the target layout while the regularization term, weighted by Areg pre- vents attention from leaking outside the target regions. The summation over y E aggregates contributions across multiple attention layers.
We define our total objective as a weighted sum of four loss components:
Lotal
AoEnoise + La + X2 (CT + Lttiit Lvar)
(6)
4 EXPERIMENTS
4.1 TRAINING AND TEST DATA
For training, we use the FS-COCO dataset (Chowdhury et al. 2022). as the only available dataset of freehand sketches. FS-COCO augments subset of 10,000 MS-COCO (Lin et al.) 2014) images with paired freehand sketches and captions that describe the sketches. We use the split of FS-COCO from (Bourouis et al.) 2024), as it guarantees that the sketches in the test set contain subset of sketch styles not seen during training. In total, we use 9,525 samples for training and 475 for testing. For each image in the FS-COCO training set, additionally generate synthetic sketch using the method by Su et al. (2023). We conduct all analyses in this section based on the methods' performance the 475 freehand sketches.
4.2 IMPLEMENTATION DETAILS
We obtain Mgth for freehand sketches by computing the similarity between each feature patch, extracted using pretrained semantic sketch encoder (Bourouis al.) 2024), and the CLIP text embedding of object P;. A threshold of 0.5 is then applied to produce a binary mask. We set Ån Å₁ 1.0 and A2 0.1. Our batch consists of 50% freehand and 50% synthetic sketches. For freehand sketches, we set Ao = 0.0 and use captions from the FS-COCO dataset. Training restricted to the top 10% of diffusion timesteps, corresponding high-noise regimes which are known to control the overall semantic structure. During inference, only those 10% noise steps are modulated. We ablate this choice in Sec. [D.3] As the diffusion backbone for our method in this section, we use Stable Diffusion 2.1 (SD2.1) (Rombach et al.) (002)). chosen for its relatively lightweight architecture and still strong generative performance. During training, the total number of noise steps is T = 1000, and during inference, it is 7 = 50. We train on single A A100 GPU. We provide additional results with the SDXL backbone in Sec.
A.3
4.3 EVALUATION METRICS
We evaluate our method using three metrics: (i) FID (Heusel et al. 2017), to assess the visual quality and diversity of generated images against real ones; (ii) sketch-image similarity, computed via cosine similarity between sketch encoder and CLIP image embeddings, assessing consistency with the input sketch; and (iii) LPIPS (Zhang et al.) 2018b), to measure perceptual similarity between generated and reference images, capturing human-aligned visual fidelity beyond pixel-level differences..

---

Published as a conference paper at ICLR 2026
4.4 COMPARISON TO STATE OF THE ART
In this section, focus evaluation performance on freehand sketches; additional compar- isons on synthetic sketches are provided in Sec. D.2 Results of the user study where participants rank outputs from different methods are provided in Sec. A.2
4.4.1 BASELINES AND CHOICE OF THEIR OPTIMAL PARAMETER SETTINGS
We compare our approach with several widely adopted training- based conditional diffusion meth- ods: ControlNet (Zhagg et al. 2023) with two diffusion model backbones SD2.1 (Rombach et al.
2022) and SDXL. (Podell et al. 2023), ControlNext (Peng et al., 2024). T21-Adapter (Mou et al. 2023). and SG (Voynov et al.] 2023). Note that these methods were originally trained on sketch styles that match our synthetic sketches. We refer this setting as zero-shor. We also compare our method with the inference- time method FreeControl (Mo et al. 2024). Additionally, in Sec. A.5 we
compare with Kontext (bfl).
ControlNet and T2I- 221-Adpteer include control parameters that balance sketch fidelity and image real- ism. ControlNet supports a scale factor w for the conditional branch connections. For T2I-Adapter, there are two parameters Jiang et al. (2025): 8, which controls how much the conditional features are added residual connections, and т, which controls for which time steps / > T(1 - 7) the spatial guidance is added, where T' is the total number of steps used.
Since our goal to generate images with the best trade off between photorealism and alignment with the input sketch, we conduct a user study to identify the optimal parameter setting for each method on our freehand sketches. When multiple settings yield similar user rankings, we additionally con- sider quantitative metrics to select the configuration that provides the best overall performance. We found the following setting to be optimal: w 0.4 for ControlNet SD2.1, w = 0.6 for ControlNet SIXL,, and two of parameters for T21 s 0.8, v 0.2 and 0.8, 0.4. We refer the reader to Sec. C for more details, and visual comparisons among different parameter settings.
We begin by evaluating all models in zero-shot setting, using pre-trained weights without modifi- cation. To enable fair comparison, we further fine-tune each baseline on our training set using two configurations: (i) optimization with the denoising loss alone (Bq. (I). and (ii) joint optimization with both the denoising and attention supervision losses (Eq. (5) ).
4.4.2 QUANTITATIVE COMPARISON
Table T shows that our method achieves the best FID, CLIP similarity, and LPIPS sketch-to-image alignment among all baselines. This demonstrates model's stronger ability to retain semantic and structural cues from abstract sketches
For ControlNet, first perform naïve mee-tuning independently on synthetic and frechand sketches from our dataset, treating them as separate training sets. On synthetic sketches, while this leads to a slight increase in FID, both CLIP similarity and LPIPS improve, suggesting better adherence the sketch input, expected from fine-tuning. The higher FID may stem from the image quality within the MS-COCO dataset, which we use for both training and fine-tuning. However, when fine-tuning on freehand sketches, all metrics degrade. We attribute this to the lack of pixel-level alignment in freehand sketches, which limits model's 's ability to learn effective spatial guidance from them. When we add our proposed attention loss and train on a mix of synthetic and freehand sketches, FID decreases only slightly, while both CLIP similarity and LPIPS improve substantially.
For T21, 0.8, 0.2 results in better FID scores, close to those of ControlNet, but poor alignment with freehand sketches. Setting = 0.8, r = 0.4 characterizes in lower FID scores, but better alignment with input sketches. In both cases, naïve fine-tuning synthetic sketches improves the performance according to all metrics. Adding the attention loss further improves performance, with particularly pronounced effect on FID scores. These results show the importance of the proposed attention loss when targeting freehand sketch inputs.
mix of frechand and
ControlNet SDXL (zero-shot) results in images of poor quality, poorly aligned with input sketches. Finally, ControlNex et al.) (2024) (Fig. [66 generates visually appealing images but often fails to accurately follow the structural guidance provided by the sketch input.
7

---

Published as a conference paper at ICLR 2026
Method
Setup
FID
CLIPT
LPIPST
Zero-shot
135,595
1.136
0.773
Luoise only (syn)
136.821
1.141
0.771
CntrlNet SD2.1
Zhang et al.
2023
Luoise only (free)
139.872
1.042
0.789
Lnoise + Latss
135.891
1.196
0.768
T2I
Zero-shot
144.329
-0.203
0.813
Adapter
Caoise only
138.479
0.318
0.779
8 0.8 r
Mou
2023
Luoise + Cates
137.982
0.391
0.774
T2I
Zero-shot
159.816
0.213
0.819
Adapter
Luoise only
151.736
0,426
0.781
0.8 0.4 (Mou al.
2023
Luoise Luns
139.568
0.454
0.778
CntrlNet SIXXI. /Zhang al
2023
Zero-shot
174.462
0.027
0.825
CatrINext SDXL [Peng el al.
2024
Zero-shot
134.094
0,909
0.774
SG (Voynov et al. (023)
7ero-shot
137.381
1.043
0.782
FreeControl (Mo et al. 2024)
Inference-time
141.632
1.089
0.793
Ours
Full
121.973
1.291
0.739
Table Comparison on 475 test sketches from the FS-COCO dataset (Chowdhury et al. 2022) with baseline methods with different fine-tuning setups. The used metrics are summarized in Sec.
Sketch
Ours
CntriNet SD2.1
CntriNet SD2.1
CntriNet SDXL zero-shot
T21 s0.8r0.4 zero-shot
T21 s0.8:04
SG
FreeControl
zero-shot
IT+attention loss
FT+attetnion loss
A girl is sitting mar a fiinn
kite
the on hores
A banch in
WWW
Figure 3: Visual comparison between method and baselines in zero-shot setting (using weights of pretrained models) and with the ones fine-tuned with the proposed attention loss on mix of freehand and synthetic sketches.
4.4.3 QUALITATIVE COMPARISON
As shown in Figs. 6 and our method produces photorealistic images that faithfully capture the key structural elements of the input sketches. ControlNet SD2.1 frequently fails in the zero- shot setting on freehand sketches, often generating implausible images that do not adhere to the input guidance. While ControlNet SDXL might better adhere to the sketch input than its SD2.1 counterpart, its outputs tend to be desaturated, blurry, or exhibit a cartoon-like appearance. Lower values of the weighting factor would yield higher visual quality, but at the cost of reduced sketch adherence, as illustrated in Fig. [4] Similarly, T2I variants with the user-study-base parameter settings overall follow sketch inputs, but produce washed-out, often cartoonish images. Higher values of r lead to better sketch adherence, but lower image quality.
Fine-tuning timeetuning on mix of synthetic and freehand sketches using our attention loss leads to improved visual adherence to the input sketches and more vivid color palettes compared to zero-shot coun- terparts. However, the results remain less consistent and exhibit a higher prevalence of artifacts compared to our full method.
4.5 ABLATIONS
4.5.1
ROLE OF SKETCH REPRESENTATION
First, ablate the choice of sketch representation used in modulation network. A common baseline is to encode sketches using the VAE encoder shared with image inputs, as done Con- trolNet (Zhang et al. 2023). In this setup, introduce separate branch for VAE-based sketch
8

---

Published as a conference paper at ICLR 2026
Sketch
(full model)
Fixed finetuned
encoder
encoder
VAE encoder
Sketch directly
195.0
FID Score
139..
140.0
1251
1357
CLIP Score
1.29
109
1.05
Fixed skatch encader
Oursi
CLP finatuned encoder
VAE
Sketch
Figure 4: Quantitative evaluation of the role of sketch representation. Please refer to Sec. 4.5.1 for the details.
features. This branch mirrors the architecture of the noise and latent branches but employs its own set of weights. As shown in Fig. 4 this leads to significantly higher FID and lower CLIP similarity scores, indicating degraded performance. In this configuration, the modulation network underper- forms compared to ControlNet (Tab. I
Next, experiment with directly using sketches as input to the modulation network, following the approach of Haa et al. (2023). Unlike our method, their modulation network employs a single- branch UNet (Ronneberger al. 2013). where all inputs are concatenated before being processed. In contrast, we find that using separate downsampling branches for each modality leads to improved performance. In this experiment, sketches are encoded using a dedicated branch consisting of pro- gressive downsampling and channel expansion via convolutional layers with SiLU activations, fol- lowed by several dime-conditioned convolutional blocks. As shown in Fig. 4 this setup results in the worst performance, suggesting that it fails to fully leverage the information encoded in sketches.
Finally, Fig. underscores the importance of our sketch encoder fine-tuning strategy. Even without fine-tuning, our model outperforms all baselines (Tab. []. However, the original CLIP encoder (Radford et 202T). even with fine-tuning, fails match our approach.
4.5.2 ROLE OF EACH OF THE LOSSES
We first ablate the role of attention loss by training variant of our model with A = 1 and X2 0 for both synthetic and free-hand sketches, effectively reducing the objective a standard diffusion loss with regularization terms without attention supervision. As shown in Fig. 3 removing the attention loss leads to a modest improvement in FID score, but significantly reduces sketch-image alignment.
Removing Evar degrades both image quality, as indicated by higher FID scores, and sketch-image image alignment, as reflected in lower CLIP similarity (Fig. (5). Thus, this loss results in higher expressivity of the model, facilitating the learning process.
Ours
WO attention WO variance
Ours (full model)
WO attention WO varlance
Sketch
(full model)
loss
loss
Sketch
loss
loss
FID
12187
11763
"A boat moving in river near big ban tower bridge."
big tree and near hillside."
125
120
115
100
0.5
Oura full
Wo var
wo alm
cow is running and dog is side."
"Dog playing garden."
Figure 5: Qualitative and quantitative evaluation when we (i) remove the attention loss Laun (Fq. 5
'WO attention loss', and (ii) when remove Evar (Eq. (3. "WO variance loss". Please refer to Sec. 4.5.2 for the detailed discussion. Captions are taken as-is from the FSCOCO dataset.

---

Published as a conference paper at ICLR 2026
5 CONCLUSIONS
Our main contribution lies shifting the focus of pixel-aligned spatial conditioning to semantic- generation, which emphasizes what is in the scene and where it is located. We address the under-explored problem of generating images from real freehand scene sketches, which often ex- hibit abstraction and distortion. We show how to extract semantic features using an appropriate encoder and condition a diffusion model on them without altering the backbone architecture. Our design, including our proposed attention loss, allows us to effectively train on freehand sketches, consistently improving performance across baseline conditioning methods.
ACKNOWLEDGMENTS
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grant No.: the NSERC Fonds de recherche du Québec Nature et technologies (FRQNT) NOVA Grant No. 314090, FRQNT team grant No. 361570, and gift from Adobe.
("Modelling and animation via intuitive input"),
REFERENCES
Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a-scene: Extracting multiple concepts from a single image. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-12, 2023a.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Transactions Graphics (TOG), 42(4), 2023b.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, al. ediff-i: Text-to-image -mage diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.013244 2022.
FLUX.1 Kontext - Black Forest Labs Flux kontext [Accessed 25- 25-11-2025).
bfl.ai.
https://bfl.ai
Ahmed Bourouis, Judith E Fan, and Yulia Gryaditskaya. Open vocabulary semantic scene sketch understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4176-4186, 2024.
Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04227, 2024.
Li-Wei Chen. http:://hhhhhooooo////coesss/sss github. com http://wwwcccceeoo////////eaaaoa
2023. URL https://
Wengling Chen and James Hays. Sketchygan: Towards diverse and realistic sketch to image syn-
thesis. In CVPR, 2018.
Shin-I Cheng, Yu-Jie Chen, Wei-Chen Chiu, Hung-Yu Tseng, and Hsin-Ying Lee. Adaptively- realistic image generation from stroke and sketch with diffusion model. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.
Pinaki Nath Chowdhury, Aneeshan Sain, Ayan Kumar Bhunia, Tao Xiang, Yulia Gryaditskaya, and Yi-Zhe Song. Fs-coco: towards understanding of freehand sketches of common objects in context. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel. October 23-27, 2022, Proceedings, Part VIII, pp. 253-270. Springer, 2022.
Sandra Zhang Ding, Jiafeng Mao, and Kiyoharu Aizawa. Training-free sketch-guided diffusion with latent optimization. arXiv preprint arXiv:2409.0033 2024.
Merlin Donald. Origins of the modern mind: Three stages in the evolution of culture and cognition.
Harvard university press, 1993.
Mathias Eitz, James Hays, and Marc Alexa. How do humans sketch objects? ACM Transactions on Graphics (TOG), (4), 2012.
10

---

Published as a conference paper at ICLR 2026
Chengying Gao, Qi Liu, Qi Xu, Limin Wang, Jianzhuang Liu, and Changqing Zou. Sketchycoco: Image generation from freehand scene sketches. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5174-5183, 2020.
Cusuh Ham, James Hays. Jingwan Lu, Krishna Kumar Singh, Zhifei Zhang, and Tobias Hinz. Mod- ulating pretrained diffusion models for multimodal image synthesis. In ACM SIGGRAPH 2023 Conference Proceedings, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Aaron Hertzmann. Comparing perspective in drawings, photographs, and perception. Art & Percep- fion, 1, 2025.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: cre- ative and controllable image synthesis with composable conditions. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Xun Huang, Ming-Yu Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV, 2018.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Lifan Jiang, Shuang Chen, Boxi Wu, Xiaotong Guan, and Jiahui Zhang. Vidsketch: Hand-drawn sketch-driven video generation with diffusion control. arXiv preprint arXiv:2502.0110/ 2025.
Rui Jiang, Guang-Cong Zheng, Teng Li, Tian-Rui Yang, Jing-Dong Wang, and Xi Li. A survey of multimodal controllable diffusion models. Journal of Computer Science and Technology, 39(3): 509-541 2024.
Jongejan, H. Rowley, Kawashima, Kim, and Fox-Gieg, Quick, draw! dataset, 2017. URL Accessed: 19.
Mttpss//ggithnn http:///hh..bb.coooogoooegeeeree 11-2025.
Subhadeep Koley, Ayan Kumar Bhunia, Deeptanshu Sekhri, Aneeshan Sain, Pinaki Nath Chowd- hury, Tao Xiang, and Yi-Zhe Song. It's about your sketch: Democratising sketch control in diffusion models. In CVPR, 2024.
Tencent ARC Lab. (2-addaperr-keechhssdk---- https::/hugginggacc. co ctt.r/aacc..///...eeetet/////// t2i adapter- seatthrr-seech--eee sdx1 2023. Accessed: 2025-09-24.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV. Springer, 2014.
Runtao Liu, Qian Yu, and Stella X Yu. Unsupervised sketch photo synthesis. In ECCV. Springer, 2020.
Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-fre spatial control of any text-to-image diffusion model with any condi- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7465-7475, 2024.
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for exxt-to-image diffusion models. arXiv preprint arXiv:2302.08453. 2023.

---

Published as a conference paper at ICLR 2026
Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Bhin-Chang Yang, and Jiaya Jia. Controlnext: Powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.06077. November 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015; 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-244-2000 23.-241. Springer, 2015.
Anceshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Clip for all things zero-shot sketch-based image retrieval, fine-grained or not. CVPR, 2023.
Zhuo Su, Jiehua Zhang, Longguang Wang. Hua Zhang, Zhen Liu, Matti Pietikäinen, and Li Liu. Lightweight pixel difference networks for efficient visual representation learning. IEEE Trans- actions Pullern Analysis and Machine Intelligence, 45(12), 2023. doi: 10. 10.1109/TPAM.2022. 3300513.
Wenqiang Sun, Teng Li, Zehong Lin, and Jun Zhang. Swat latent initialization for control- lable image generation. arXiv preprint arXiv:2401.16155 2024.
Barbara Tversky. What do sketches say about thinking. In 2002 AAAI Spring Symposium, Sketch Understanding Workshop, Stanford University, AAAI Technical Report SS-02-08, volume 148, pp. 151. 2002.
Gizem Esra Unlu, Mohamed Sayed, Yulia Gryaditskaya, and Gabriel Brostow. Groundup: Rapid sketch-based 3d city massing, In Proceedings of the European Conference on Computer Vision (ECCV), July 2024.
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guide text-to-image diffusion mod- cls. In ACM SIGGRAPH 2023 Conference Proceedings, 2023.
Qiang Wang, Di Kong, Fengyin Lin, and Yonggang Qi. Diffsketching: Sketch control image syn- thesis with diffusion models. arXiv preprint arXiv:2305.18812 2023.
Sheng- Yu Wang, David Bau, and Jun Yan Zhu. Sketch your GAN. In ICCV, 2021.
Zhenbei Wu, Haoge Deng, Qiang Wang, Di Kong, Jie Yang, and Yonggang Qi. Sketchscene: Scene sketch to image generation with diffusion models. In 2023 IEEE International Conference Multimedia and Expo (ICME). IEEE, 2023.
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creating high-quality 3d assets. ACM Transactions Graphics (TOG), 43(4), 2024a.
Lvmin Zhang. sdDl-ccontoolnet httpps//aacqinn/aaee../////////
sd-cont rol-contollin net scr bb le 2023. Accessed: 2023-09-24.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
12

---

Published as a conference paper at ICLR 2026
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018a.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018b.
Tianyu Zhang, Xiaoxuan Xie, Xusheng Du, and Haoran Xie. Sketch-guided scene image generation. arXiv preprint arXiv:2407.06480 006699 2024b.
13

---

Published as a conference paper at ICLR 2026
A
ADDITIONAL EVALUATIONS
A.1 ADDITIONAL VISUAL RESULTS
Additional visual results, comparing our method and baselines, provided in Figs. and
Sketch
ControlNet 2ero-shot
ControlNet FT4altention loss
SontrolNet 2810 shot
60.2 zero-shot
FT+ attettion loss
zero-shot
FT+athetrion loss
Seed
Two giraffes standing the sco.
Train going track.
girls playing with friabe dise.
Seed
man is flying sky.
veed
a berchh in. the garden
Seee
Figure 6: Qualitative results comparison between our method and baselines. Please see Sec. 4.4.2 for the detailed discussion. Sketch and text are both passed as inputs.
A.2 USER PREFERENCE STUDY
We conducted a user study involving 23 participants to subjectively evaluate generation quality. We compared our method against the best-performing baselines: ControlNet SD2.1, ControlNet SDXL, and T2I-Adapter. The sketches were randomly selected from the entire test set for each participant. Each participant was asked to evaluate 20 pairwise comparisons: ours against one of the baselines. The same sketch was not shown twice to users. We asked participants to pick an image according to each of the following criteria: (i) Which of the images looks photorealistic (i.e., most like
14

---

Published as a conference paper at ICLR 2026
Skotch
Ours
ControlNett shrt
ControlNet
Controlivet SDX. z010 shot
s0ar zero shot
T2 zero-shat
FTtattontoo loss
Flrattetiion loss
FTrahstrian loss
Airplane standing the sirport.
Three people are walking halding umbrr ambbrellas.
girl sitting an korse.
Seed1
A building with clock
Secee
people sitting beside the foed table open ground.
Figure 7: Qualitative results comparison between our method and baselines. Please see Sec. 4.4.2 for the detailed discussion. Sketch and both passed as inputs.
a real photograph)? (ii) Which image is more similar to the sketch terms of layout and overall structure? (ii) Which image the better trade-off between photorealism and similarity to the sketch?
Photorealism (%)
Sketch similarity (%)
Best trade-off (%
Baseline
Ours
Base
Und.
Ours
Base
Und.
Ours
Base
Und.
SDXL T2I s=0.8 r=0.2
47.2
34.9
17.9
46.4
37.1
16.5
66.0
20.6
13.4
SDXI. T21 s=0.8 7=0.4
47.4
39.2
13.4
55.0
35.8
9.2
71.0
22.0
7.0
ControlNet SDXI.
40.4
45,2
14.4
54.0
9.0
54.4
32.2
13.3
ControlNet SD2
70.0
21.2
8.8
64.8
30.8
4.4
88.8
10.0
1.2
Table Pairwise user study preferences. Please see Sec, A.2
for the details.
As shown in Tab. 2 our method is consistently preferred across all baselines for the third question ("best trade-off"). While it trails slightly behind ControlNet SDXL. in perceived photorealism
15

---

Published as a conference paper at ICLR 2026
Method
Setup
FIDI
CLIPT
LPIPS,
Zero-shot
174.462
0.027
0.825
CntrlNet SDXL
#2
ControlNet branch LoRA
177.243
0.046
0.813
Zhang
2023
ControlNet branch LoRA
#3
189.566
0.043
0.779
main branch LoRA
#4
Ours S
Modulation network training
121.973
1.291
0.739
#5
Ours SDXL
Modulation network training
117.025
1.331
0.708
Modulation network training
#6
Ours SDXL
138.213
1.162
0.759
+ backbone LoRA fine-tuning
Table 3: Comparison on 475 test sketches from the FS-COCO dataset (Chowdhury et al. 2022 averaged over three seeds. (#4) Ours SD2. / is the main model used throughout the paper, where train only the modulation network and the last three layers of the sketch encoder with all the proposed losses as described in the main paper. All remaining variants use the SDXL backbone. (#5) Ours SDXL (Modulation network training) differs from (#4) Ours SD2.I (Modulation net- work training) only in the backbone, while the training setup remains the same. (#6) Ours SDXL (Modulation network training+bukkbore LoRA fine-tuning adds additional LoRA fine-tuning of the backbone SDXL model.
For ControlNet SDXL, LoRA fine-tuning applied either to the ControlNet branch alone (#2) or to both the NontrolNet and main diffusion branches (#3). The used metries are summarized in Sec. 4.3
Analysis provided in Sec.
likely due to the latter's stronger diffusion backbone is strongly favored over ControlNet SD2.1, which shares the same backbone. Our method is also consistently rated as producing images that better match the input sketches.
A.3 TRAINING WITH SDXL BACKBONE
In the main paper, we use the SD2.1 backbone with our method due to its lower computational requirements and faster iteration cycles. To demonstrate that our method is backbone agnostic, we additionally train our pipeline with the SDXL backbone (Tab. Fig. 8). Compared to SD2.1, the SDXL backbone employs a substantially larger and more expressive architecture, offering improved generative fidelity but requiring significantly more computational resources.
We previously showed that our method with the SD2.1 backbone outperforms all alternatives. Here we show that using a stronger backbone further improves our method across all three metrics: FID, CLIP, and LPIPS (Tab. #5 vs. #4). Switching from SD2. SDXL reduces FID from 121.973 to 117.025, increases CLIP sketch-image similarity from 1.291 to 1.331, and decreases LPIPS from 0.739 to 0.708. These results demonstrate that noise-modulation strategy, together with atten- tion supervision, generalizes effectively to larger backbones without requiring any fine-tuning of the backbone itself. Consequently, our method serves an efficient, lightweight conditioning mecha- nism.
We additionally experiment with applying LoRA fine Ane-tuning to the SDXL backbone alongside train- ing our modulation network, which serves as a conditioning mechanism. This approach results in worse performance than training the modulation network alone (compare the last two rows in Tab. We hypothesize that this is due to slower training dynamics the number of trainable parameters increases. All baselines in this table are trained for the same number of iterations (5K).
A.4 CONTROLNET WITH LORA FINE-TUNING
As our method introduces a lightweight conditioning mechanism that optimizes only small number of parameters (9.4M), compare it against ControlNet under LoRA fine-tuning setup. Note that we begin with the ControlNet variant whose control branch has already been fine-tuned on synthetic sketches generated programmatically from reference images. fine-tuning the full ControlNet branch requires training 1.5B parameters. this setting, we only use the original diffusion loss, but train on combination of freehand and synthetic sketches.
As shown in Tab. 3, LoRA performs poorly in this setting. Applying LoRA only to the ControlNet branch slightly improves CLIP similarity (0.027 0.046) but results in worse FID (174.462
16

---

Published as a conference paper at ICLR 2026
177.243). Extending LoRA to both the ControlNet and the diffusion backbone further degrades FID to 189.566, while CLIP similarity remains low (0.043). Applying LoRA only to the ControlNet branch requires training 12M parameters. These observations are consistent with the experiments reported in the main paper, where fine-tuning all weights of the ControlNet (SD2.1) branch on combination of freehand and synthetic sketches proved suboptimal. These experiments show that naive fine-tuning strategies with freehand sketches degrade the visual quality of the generated images.
Both Tab. and Fig. demoonstrate that our conditional mechanism, implemented via the proposed modulation network and combined with the attention supervision loss, achieves superior results regardless of the backbone. Furthermore, using a more powerful backbone leads to improved per- formance. Ours (SDXL.), similarly to Ours (SD2. 1), provides favorable tradeoff between sketch adherence and visual realism.
Implementation details In all LoRA experiments, we use a rank of 16, alpha of 32, and dropout of 0.1, with a learning rate of 0.0001. Training is performed for 5K iterations with a batch size of 128 on A100 GPUs. Both training and evaluation are conducted on the same dataset as in the main paper, consisting of a mixture of freehand and synthetic sketches (see Sec. 4.1
A.5 COMPARISON WITH FLUX.1 1 KONTEXT [DEV]
Additionally, we compare our method with the recent universal image editing model Fluxkontext (bi)), based on FLUX. Although this model is not specifically trained for sketch-to-image conver- sion, it benefits from training on a massive dataset. We prompted the FluxKontext model with "Convert sketch to a photorealistic image of concatenated with the same prompt pass to our model. We experimented with set prompts for the Flux Kontext model and empirically found that this one yielded the best results.
As shown in Fig. 9 our method demonstrates clear advantages over the state-of- thetthrrrrttt universal image editing model Flux.1 Kontext [dev]. Our model produces more photorealistic output, better follows input sketches, and remains robust even with distorted or cluttered sketches. In the bear and plane examples on the left in Fig. 9 we can see that, unlike Flux. Kontext [dev], our method follows the sketch well and produces variations in visual details and colors across different seeds. In the dog example, Flux Kontext fails localize the dog in the grass.
Quantitatively, we achieve a lower FID (121.973 vs. 132.487) and a higher CLIP sketch image sim- ilarity score (1.291 vs. 1.226), validating both the superior realism and the stronger sketch alignment for our method.
A.6 GENERALIZATION TO QUICKDRAW DATASET
To assess generation beyond scene-level sketches, we also evaluate our method on single-object sketches from the QuickDraw dataset Jongejan et al. (2017). To illustrate how our method per- forms on single-object sketches relative to specialized state-of-the-art approaches, we include a visual comparison with results from AboutYourSketch [3], recent method specifically designed for single-object sketches and built on the SDI.5 backbone. As shown in Fig. [0] our approach achieves visual quality comparable to the specialized single-object method YboutYourSketch Ko- ley et al. (2024), despite never being trained on isolated-object data. Our model preserves object shape and produces photorealistic textures on par with the dedicated baseline, while maintaining the broader flexibility needed for complex multi-object scenes. For our method, we use the prompt: "A photo of <ca <category>> For SkurScch,,,, the visual results are taken directly from the paper. A numerical comparison was not possible due to the unavailability of the baseline's code.
A.7 COMBINING CONTROLNET WITH PAINT- PAINT-WITH-WORD GUIDANCE (BLLJJ ET AL. 2022
A public implementation CChen 2023) combines ControlNet with paint-with-words guidance (Balaji ctt al. 2022). which uses user-dra uiser-drawn masks to steer tross-attention during inference to add semantic guidance. Fig. 11 shows that the performance is dependent the setting of control parameters
17

---

Published as a conference paper at ICLR 2026
for each object category, while our method produces consistent results, without the need for the segmentation masks and object-specific parameter tuning.
18

---

Published as a conference paper at ICLR 2026
Inputs
(SD2.1)
(SDXL)
(SDXL)
ControlNet (SDXL) ControlNet (SDXL) ControlNet (SDXL)
the backbone
LoRA
ControlNet
branches
Two giraffes standing
Train going on the track.
Two girls are playing with frisbee disc.
flying sky
Bench the Garden
Figure 8: Qualitative comparison our method using the SD2.1 backbone, the SDXL backbone, and the SDXL backbone with LoRA fine-tuning of the backbone. We also compare against Control- Net (SDXL backbone) in three configurations: zero-shot, LoRA applied to the control branch, and LoRA applied to both the diffusion and control branches. All methods are conditioned the input sketches and textual captions shown in the figure. The results are shown for two different seeds.
19

---

Published as a conference paper at ICLR 2026
Input
Ours (SD2.1)
Flux Kontext
Input
Ours (SD2.1)
Flux Kontext
Airplane
Giraffes are eating the tree leaf.
flying in the sky.
a bear in the forest
Dog playing in the garden.
Figure 9: Qualitative results comparison between method and Flux. I Kontext [dev]. Our method conditioned on sketches and textual captions, shown in the figure. We prompted the FluxKontext model with "Convert sketch photorealistic image of ...", where instead of "..." we the same prompt as we pass to our model. The results are shown for two different seeds.
"It's all about your sketch"
Ours
Figure 10: visual comparison between our method and specialized single object sketch-based image generation method Koley et al. (2024) on the sketches from the QuickDraw dataset Jongejan al. (207))
20

---

Published as a conference paper at ICLR 2026
ControlNct paint
Inputs
Parameter settings
Results
Parameter settings
Results
Results
dunona 6.5,
curonng 1.2,
Saltroouu15
Aatmmoon, incuttatiss1 ehottSrupen lake 1.5 beat
mountotos hallirsse (o boot.2
A dlgitaa pointing a hoo fromn lalerneor mountain moon and aurova A boot is intherondde of the lake.
digital pointing plaitalpottting ofa half frome laterneo mountsins moon and aurons bout in uniteenddle the lakke...
Figure 11: A visual comparison between our method and public implementation Chen the method that combines ControlNet with paint-with-words guidance Balaji et al. (0022
2023 of
21

---

Published as a conference paper at ICLR 2026
B IMPLEMENTATION DETAILS
B.1 MODULATION NETWORK
The detailed design of our modulation network is shown in Tab.
Our sketch encodings have di- The VAE latent representations
mensions 512 x 14 x 14, where 512 denotes the number of channels. of size x 128 128, where the number of channels.
Path/Block
Layer/Operation
Input Ch.
Output Ch.
Spatial Change
DoubleConv
512
256
Sketch Path
DoubleConv
256
128
Interpolate (bilinear)
128
128
Match latent/noise size
DoubleConv
MaxPool2d (2)
DoubleConv
16
32
Noise Path
MaxPool2d (2)
32
32
/2
DoubleCony
32
64
MaxPool2d (2)
DoubleConv
64
128
DoubleConv
4
16
MaxPool2d (2)
12
DoubleConv
Latent 2; Path
MaxPool2d (2)
32
32
12
DoubleConv
32
64
MaxPool2d (2)
64
64
12
DoubleConv
Fusion
Concat [s2, p4, 14]
384
DoubleConv (fusion)
384
256
Conv ConyTranspose2d (upl)
256
128
x2
DoubleConv (up.com convl)
Conv TonnTraasppse2d (up2)
64
32
x2
Upsampling
DoubleConv (op_conv2]
32
16
Conv' Conn'rranposeed (up3)
16
8
x2
DoubleConv (up_com (up_conv3)
Final
Conv2d (final)
torch.chunk
8
4+4
Table Architecture of the proposed modulation network.
B.2 CONTROLNET AND T2I ADAPTER FINE-TUNING
We start from pretrained sd22--controinet-serbbbb Zhang 2023 and finetune it on our dataset Sec. Our training objective combines the standard diffusion loss Eq. ( with an attention align- ment loss Eq. (5). We extract attention maps from the U-Net decoder at resolutions 8x8, 16x16, 32x32 and supervise them with Mgrth using Eq. (5). Training uses Adam (Ir=le-5) for 5k itera- tions on single A 100 GPU, with batch size of 32 and batch composition of 50% freehand and 50% synthetic sketches.
We 22-aadapter-sketch-ssttt adapter. We apply the same attention supervision strategy by extracting cross-attention maps from decoder layers at 8x8, 16x16, 32x32 resolutions. We use identical hyperparameters (Adam Ir=le- 5, batch size 32) and train for 5k iterations.
Lab (2033 pre-trained model for the fine-tuning of the T2I
22

---

Published as a conference paper at ICLR 2026
C SELECTING PARAMETERS IN CONTROLNET AND T21
As mentioned in Sec. 4.4.1 ControlNet Zhang et al. (2023) and T21-Adapter Mou et al. (2023)
include control parameters that balance sketch fidelity and image realism. ControlNet supports setting a layer-dependent scale factor w for the conditional branch connections. In practice, it is sufficient define w for the top layer, as values for lower layers are automatically inferred based on the layer's depth.
For T2I-Adapter, we experiment with two parameters Jiang al. (2025): s. which controls how much the conditional features are added in residual connections, and т, which controls for which time steps T(1 - 7) the spatial guidance is added, where 7 is the total number of steps used.
C.I SELECTING THE BEST SETTING
To select the best settings of control parameters, we ask a user to choose among a few possible image options obtained with different parameter settings. 7 participants took part in this user study. The results of the user study summarized in Fig. 12
study, where for each of the baselines
Visual results are shown in Figs. 13 to 16 while numerical results are shown in Tab, 5
Average Preferences
Question
ControlNee_ Photorealism
ControlNet 5D2: Sketch Similenty
ControlNet Best Trade-off
10
28
ControlNee Photorealism
ControlNeet SDXL: Skettch Similarity
Controlivet SOXLL Best Trade
2
Parameter Values
Parameter Volues T2): Skatch Similarity
Parameter Values
Photorcalism
T2l: Trade off
20
Rid a
45.5 2014
auu
add
Kta
Rarametor Values
Parameter Volues
Paametee Values
Figure 12: Parameter preferences by model by question.
23

---

Published as a conference paper at ICLR 2026
Baseline
Param
FID 4
CLIP stm
LPIPS
w0.2
141.46
0.911
0.780
w0.4
138.62
0.927
0.774
ControlNet SD2
s0.6
135.60
1.136
0.773
w0.8
137.35
0.765
0.773
w1.0
143.42
0.668
0.786
w1.2
147.53
0.468
0.791
w0.2
162.35
0.032
0.822
w0.4
174.46
0.027
0.825
ControlNet SDXL
w0.6
196.35
-0.412
0.830
w0.8
236.58
-0,455
0,846
w1.0
219.09
-0.222
0.836
w1.2
243.59
-0.039
0.831
w0.2
131.58
0.841
0.772
w0.4
134.09
0.909
0.774
ControlNext SDXL
w0,6
149.19
0.279
0.781
w0.8
192.92
-0.357
0.796
w1.0
232.27
-1.036
0.843
w1.2
237.38
-0.934
0.837
s0.8 0.2
144.33
-0.203
0.813
s0.8 0.4
159.82
0.213
0.819
s0.8 0.6
165.52
0.057
0.827
T21
$1.0 0.2
149.13
0.132
0.821
s1.0 0.4
152.41
0.213
0.820
$1.0 0.6
144.05
0.525
0.809
Table 5: Detailed performance comparison of all baseline models with different parameters. The settings highlighted in bold are the versions selected for the main paper.
24

---

Published as a conference paper at ICLR 2026
Sketch
0.0
0.2
0.4
0.6
0.8
1.0
sood
9200 2
sirplane the runway is waiting take off from the airport, control background.
S000 2
Man posing behind the fence front a giraffe in background.
A woman sitting on a wooden bench in park with her dog on the left side under the bench.
poas
A cow in a field walking while dog is lying down.
soed
seod 2
big and bench the hillside
Figure 13: Visualization of image generated with ControlNet SD2 for different values of control parameter. Each row shows results for specific sketch and text prompt using random seeds. Text captions are shown below each sketch group.
25

---

Published as a conference paper at ICLR 2026
Sketch
0.0
0.2
0.4
0.6
0.8
1.0
sood
3200 2
sirplane the runway is waiting take off from the airport, control the background.
S00d 2
Man posing behind the fence front a giraffe in background.
p000 2
A woman sitting on a wooden bench in park with her dog on the left side under the bench.
peas
A cow in a field walking while dog is lying down.
soed
s00d 2
big and bench the hillside
Figure 14: Visualization of image generated with ControlNet SDXL. for different values of control parameter. Each row shows results for specific sketch and text prompt using random seeds. Text captions are shown below each sketch group.
26

---

Published as a conference paper at ICLR 2026
Sketch
s0.8 r0.2
r0.
s0.8 r0.6
s1.0r0.2
s1.0r0.4
#1.0r06
xocd
soced
sirplame runwyy
airport, background.
Sp0E 2
Man behind the fence front giraffe background.
ROCE 2
woman sitting on wooden bench in park with her dog on the left side under the bench.
cow in a field walking while a dog lying down.
OOTTER
secd
big
Figure 15: Visualization of image generated with T2I with SDXL model for different values of control parameters. Each row shows results for a specific sketch and text prompt, generated using two random seeds. Text captions shown below each sketch group.
27

---

Published as a conference paper at ICLR 2026
Sketch
0.0
0.2
0.4
1.2
airplane the runway waiking to Lake off from the sirport, lower control in the backgrroudd.
posing
girafie background.
A woman sitting on wooden bench park with her dog on the side under the bench.
Z. pooce
A cow in field walking while dog is lying down.
big beach hillside
Figure 16: Visualization of image generated with ControlNext for different values control param- eter. Each shows results for specific sketch and text prompt using two random seeds. Text captions are shown below each sketch group.
28

---

Published as a conference paper at ICLR 2026
D ADDITIONAL ABLATIONS
D.1 TRAINING ON FREEHAND SKETCHES ONLY
To evaluate the importance of training both synthetic and freehand sketches, train variant of our model using only freehand sketches optimized with the full objective in Eq. (6). As shown in Tab. 6 it results in worse performance freehand sketches across all metries.
Method
FID +
CLIP
LPIPS +
Freehand only
142.27
0.955
0.750
Ours (full)
121.973
1.291
0.739
Table 6: Ablation: Comparing training freehand sketches alone versus training combination of freehand and synthetic sketches, with evaluation performed on frechand sketches from the FS- COCO dataset Chowdhury et al. (2022).
D.2 TRAINING ON SYNTHETIC SKETCHES ONLY
We additionally investigate variant trained exclusively on synthetic sketches, denoted as Ours (syn- thetic). This ablation allows us to evaluate whether our architecture can generalize when restricted to sketches with pixel-aligned ground-trume ttrdttttuthhae images.
D.2.1 PERFORMANCE ON SYNTHETIC EDGE-ALIGNED SKETCHES
As shown in Fig. 17 Ours (synthetic) achieves performance comparable state-of-the-art edge- conditioned methods such as ControlNet (Zhang et al. (023) and T2I-Adapter (Mou et al.] (023). This indicates that modulation network and aitention-based supervision are effective in capturing the sketch structure when constrained to just edge maps.
Ours SD2.1 (synthetic)
Syn sketch
ControlNet SD2.1
ControlNet SDXL
T2I SDXL
lody is sitting on the bench with dog aside
Man showing the baseball bai
Few sheop are eating grass on a mountain
Train going on the track
Figure 17: Qualitative results comparison between our method, our method trained on only synthetic sketches, Controlnet, and T2I adapter
Quantitatively, our synthetic variant outperforms both ControlNet SD2.1 and SDXL across all met-
rics, achieving lower FID (106.74 vs. 124.57 / 112.69) and LPIPS (0.423 0.458 / 0.432), as
29

---

Published as a conference paper at ICLR 2026
well higher CLIP similarity (1.280 1.247 / 1.286). While T21-Adapter SDXL achieves the best overall scores, our method remains competitive, particularly in FID and perceptual similarity, demonstrating the effectiveness of our approach using only synthetic sketches.
Method
FID L
CLIP 1
LPIPS
ControlNet SD2.1
124.57
1.247
0.458
ControlNet SDXL
112.69
1.286
0.432
T21-Adapter SDXL
102.43
1,383
0.379
Ours (synthetic)
106.74
1,280
0.423
Ours (full)
109.08
1.193
0.484
Table 7: Quantitative comparison of our method trained synthetic sketches (Ours (synthetic)) against baselines trained similarly, evaluated on synthetic sketches, as the ones shown in Fig. 17 The best results are highlighted in bold, and the second-best results underlined.
D.2.2 PERFORMANCE ON FREEHAND SKETCHES
Ours SD2.1 (synthetic)
Freehand sketch
Ours SD2.1
ControlNat SD2.1
ControiNet SDXL
a lady is sitting on the
banch with dog aside
Main showing the baseball ball
Few shoop are nating grass mountain
Train going on the
track
Figure 18: Qualitative comparison between our method, our method trained on only synthetic sketches, ControlNet, and T2I Adapter tested on freehand sketches.
We further evaluate this model freehand sketches test its robustness beyond pixel-aligned inputs. As summarized in Tab. g and illustrated in Fig. [8] training solely on synthetic sketches leads to a clear drop performance compared our full model: CLIP similarity falls 0.817 and LPIPS increases to 0.793. This suggests that while synthetic training suffices for edge-like inputs, exposure to genuine freehand sketches is essential to handle abstraction and distortion reliably. The qualitative results in Fig. conffiim this gap: Ours (synthetic) often captures coarse scene layout but struggles with object details and proportions, whereas the full model better respects both semantics and realism.
D.3 NUMBER OF NOISE STEPS
We first evaluate the noise steps on which our modulation network is trained. We experiment with 10%, 20%. 30%, 40% and 50% of timesteps corresponding to high noise regimes. However,
30

---

Published as a conference paper at ICLR 2026
Method
FID
CLIP f
LPIPS
ControlNet SD2.1
135.59
1.136
0.773
ControlNet SDXL
174.46
0.027
0.825
T2l-Adapter SDXL
159.82
0.213
0.819
Ours (synthetic)
132.17
0.817
0.793
Ours (full)
121.97
1.291
0.739
Table 8: Quantitative comparison of our method trained on synthetic sketches only (Ours (synthetic)) with edge-conditioned baselines tested on freehand sketches
be seen Fig. 19 as we train on more time steps, both FID and CLIP scores gradually degrade, over-constraining the model. A similar observation was reported in the T21- T21-Adapter paper Mou al. (2023), which also emphasized focusing training on high-noise regimes. Our choice of restricting training the top 10% of noise steps not only improves performance but also leads to faster conver- gence compared to training on a broader noise range. The rest the experiments in this section are trained for 10% of the highest noise steps.
sketch 10% (Ours)
20%
30%
40%
50%
FID Score
16032 160.71
187A3
121.87
CLIP Score
130
72
ad
110
118
Ours 20%
30%
40% 50%
Figure 19: Qualitative and quantitative evaluation when training with 10%, 20%, 30%, 40%, 50% and 100% of timesteps corresponding to high noise regimes. Modulation performed for the same number of time steps the respective training. Please refer to Sec. D.3 for the detailed discussion.
D.4 CONTROLNET AND OUR MODULATION NETWORK
We experimented with integrating modulation network into ControlNet. As shown in Tab. this combination yields only marginal performance changes, likely due to the dominant influence of ControlNet's s conditional branch.
Method
Sctup
FID.
CLIP
LPIPS
Zero-shot
135.595
1.136
0.773
Lacise only (syn)
136.821
1.141
0.771
CntrINet SD2.1 (Zhagg et al. 2023
Lnvise only (free)
139.872
1.042
0.789
Emvise + Lettn
135.891
1.196
0.768
Lavise Loat Mod.
134.771
1.192
0.767
Ours
Full
121.973
1.291
0.739
Table 9: Comparison 475 test sketches from the FS-COCO dataset (Chowdhury et al. 2022
with
ControlNet versions.
31


Reviews:
[{"review_id": "5uRsaZFYXU", "reviewer": "Reviewer_pKSW", "summary": "This work focuses on the task of scene-level sketch-based photorealistic image synthesis. It proposes a new modulation network and a loss function that emphasize the semantic structure of sketch inputs thereby improving generation quality. The authors have conducted experiments on the FS-COCO dataset and compared their method against existing conditional image generation methods, such as ControlNet, T2I Adapter, and FreeControl. The experimental results demonstrate that the proposed method outperforms these baseline methods across various metrics, including FID, CLIP and LPIPS.", "questions": "(1) The authors mentioned that the sketch encoder requires finetuning. Could the authors provide further details on this process? \n\n(2) What are the corresponding text prompts used to generate the examples shown in Fig. 5?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method achieves impressive visual quality in its generated results, which is further supported by strong quantitative performance.\n2. This work addresses a key challenge in sketch-based image generation, the inherent abstraction and ambiguity of sketches, which often leads to distortion in the results of existing methods. The proposed modulation network effectively addresses this issue by emphasizing the semantic structure of sketches. Consequently, the generated images achieve a good balance between visual quality and consistency with the input sketch\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The authors state that because a sketch can be abstract and ambiguous, their method focuses on extracting its semantic and structure information. In that case, it raise the question of whether an alternative approach, such as performing sketch captioning first and then feeding the resulting text into a standard T2I model (or baseline methods used in this paper), could be viable. A discussion or comparison against such a two-stage pipeline would be a valuable addition.\n2. All experiments are conducted on the FS-COCO dataset, which contains relatively high-quality freehand sketches. To better assess the model’s robustness, it would be beneficial to evaluate its performance on datasets with more abstract or simplistic sketches, e.g., QuickDraw? This would provide greater insight into the model's generalization capabilities with respect to sketch abstractness.\n3. The paper focuses on photorealistic image synthesis. It would be interesting to see if the proposed method can generalize to other artistic styles, such as cartoons. Including a few examples or a brief discussion on the model's adaptability to different styles would broaden the perceived applicability of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o6yEAg7jRb", "reviewer": "Reviewer_L6kH", "summary": "The paper tries to tackle generating photorealistic images from freehand scene sketches and preserving structure and semantics. It augments a pretrained text-to-image model Stable Diffusion 2.1 with a lightweight modulation network that scales and shifts the predicted noise using sketch-conditioned features. A cross-attention supervision loss enforces text–spatial alignment. Training mixes human and synthetic sketches and emphasizes early high-noise timesteps to balance cost and global layout learning. Experiments show consistent gains over baselines on FID, CLIP similarity, and LPIPS.", "questions": "Could the author explain more about why must cross-attention alignment be enforced via scale/shift on predicted noise finetuning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The method handles abstract and deformable sketches for the scene-level sketch-to-photo generation. Attention supervision explicitly ties language tokens to spatial regions and the modulation head is lightweight and only active in early timesteps, which keeps computation modest. It plugs into a standard SD2.1 pipeline without re-architecting. This modularity makes the approach easy to reproduce.", "weaknesses": "The paper does not offer a mechanistic explanation or theoritical analysis for why the noise-modulation head works. Evidence is largely empirical like metric tables and ablations without probing the internal reasons. Moreover, it does not clearly position the method against mainstream fine-tuning techniques such as LoRA comparison and the core distinction from other fine-tuning techniques remains under-analyzed. \n\nThe paper does not clearly articulate the mechanistic between the noise-modulation head and the target attention maps like why cross-attention alignment must be enforced via scale/shift on the predicted noise. The necessity of using target attention maps through this specific noise-space finetuning pathway remains not solid and reasonable.\n\nThe backbone choice is not state-of-the-art as the method is demonstrated on Stable Diffusion 2.1 only. There are no experiments on newer text-to-image backbones (like SDXL or FLUX). Without such results, generality are weakened and the figures presented seem relative lower quality in current phase, hurting the claim of the method effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BzkLutJxVQ", "reviewer": "Reviewer_JH8r", "summary": "The present work seeks to improve sketch-guided image generation algorithms, e.g.,ControlNet and Adapter-based models. Specifically, the authors leverage an existing sketch-focused semantic embedding model to generate features that help guide the training of a new model that includes a novel modulation network trained on a loss that estimates likelihoods of different sketch pixels belonging to different semantic categories. Notably, the authors train on a mix of automatically generated sketches from images, and freehand sketches, along with language supervision leading to the ability to represent sketches and their relationship to images at multiple levels of abstraction.\nI generally found the results quite compelling, although the improvement in metrics did not seem to always seem to qualitatively lead to images that looked much better. I found that while individual sections were well written, the organization of sections was somewhat disorganized requiring much back and forth between sections of the paper to understand the key methods and results. I elaborate more in the sections below.", "questions": "* Perhaps because I’m not deeply familiar with Ham et al., 2023, but while I understand why low variance is penalized in the scale and shift maps (S, B), could the authors motivate the L1 regularization of these feature maps too? Is this just to smoothen the training process or is there a deeper theoretical significance?\n* Could more be said about how the last 3 layers of the sketch encoder (from Bourouis et al.) was fine-tuned? \n* I think currently the ‘Attention Supervision’ part of the training pipeline is the least clear, particularly Eq 5. I would recommend a greater explanation of the Sun et al. formulation either in that section or in the Appendix.\n* I would like to better understand why training on a mix of synthetic and freehand sketches led to more vivid color palettes as the authors note.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Both theoretically and empirically, I found the attention supervision loss to be quite elegant and effective. I find this to be among the paper’s stronger contributions and a key step towards translating freehand sketches to semantically coherent images.\n* As with any good vision paper, I found the ablation experiments and comparisons to existing methods quite thorough and generally compelling.\n* While I do have issues with the general clarity of the methods overall, I found some of the overview sections (e.g., preliminaries on LDM) well executed for the amount of space taken up by the section.", "weaknesses": "* The paper is at times hard to read because of its structure. For example, the modulation network is introduced before the reader knows what the sketch features that it uses are. I recommend laying out the necessary details of all the ‘ingredients’ of a module before diving into details about the module.\n* I think there needs to be more details of the user study in the main text. I also find 23 participants to be quite a small pool.\n* While I do find the interleaved discussion of the results in the Ablation section well written, the conclusion is quite limited both in terms of length and overall takeaways.\n* While the modulation network and the attention supervision approaches are key contributions, the gains (here measured via improvements in FID, CLIP, and LPIPS) are fairly incremental with the image quality not appearing to be vastly superior. I think showing that users vastly prefer using this model relative to others would be compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "35BHElC97v", "reviewer": "Reviewer_2bmT", "summary": "### Summary\n\nThis paper introduces a method for scene sketch-to-photo synthesis. The paper argues that the previous study mainly focuses on the conversion from edge maps to photos. Thus, the proposed method is developed to convert actual freehand sketches to photorealistic images, leveraging semantic sketch features. Compared to simple ``sketch-to-photo'' methods like ControlNet and T2I Adapter, the proposed method yields more plausible results.", "questions": "I would ask the authors to clarify the novelty and merit compared to existing methods for converting free-hand (or abstract) sketches to photos.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The proposed method yields better results compred to *standard* image-to-image conversion methods like ControlNet and T2I Adapter.", "weaknesses": "### Major comments regarding the novelty\n\nAlthough the paper claims the majority of prior methods just convert edge maps to photos (this may be true to some extent; for example, ControlNet is often trained on edge maps), the conversion from abstract freehand scene sketches to photos is actively studied, as in but not limited to [a-c].\n\nCompared to those (bunch of) existing attempts, the novelty of the submitted paper is unclear. The novelty and merit (over methods inputting freehand sketches) are not adequately discussed in the related work sections, nor are they compared in the experiments.\n\n- [a] Xiang, Xiaoyu, et al. \"Adversarial open domain adaptation for sketch-to-photo synthesis.\" WACV 2022.\n- [b] Wang, Jiayun, et al. \"Unsupervised scene sketch to photo synthesis.\" ECCV 2022.\n- [c] Koley, Subhadeep, et al. \"Picture that sketch: Photorealistic image generation from abstract sketches.\" CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)