FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models
Authors: Rui Hu
Qian He
Gaofeng He
Jiedong Zhuang
Huang Chen
Huafeng Liu
Huamin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: rendered-to-real
photo-realism enhancement
image-to-image translation
diffusion model
Primary area: diffusion_based_models

Abstract: Modeling and producing lifelike clothed human images has attracted researchers' attention from different areas for decades, with the complexity from highly articulated and structured content. Rendering algorithms decompose and simulate the imaging process of a camera, while are limited by the accuracy of modeled variables and the efficiency of computation. Generative models can produce impressively vivid human images, however still lacking in controllability and editability. This paper studies photorealism enhancement of rendered images, leveraging generative power from diffusion models on the controlled basis of rendering. We introduce a novel framework to translate rendered images into their realistic counterparts, which consists of two stages: Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG). In DKI, we adopt positive (real) domain finetuning and negative (rendered) domain embedding to inject knowledge into a pretrained Text-to-image (T2I) diffusion model. In RIG, we generate the realistic image corresponding to the input rendered image, with a Texture-preserving Attention Control (TAC) to preserve fine-grained clothing textures, exploiting the decoupled features encoded in the UNet structure. Additionally, we introduce SynFashion dataset, featuring high-quality digital clothing images with diverse textures. Extensive experimental results demonstrate the superiority and effectiveness of our method in rendered-to-real image translation.

FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models
Rui Hu Qian Helt, Gaofeng He², Jiedong Zhuang',
Huang Chen?, Huafeng Liu Huamin Wang*
7Zejjann University. SStyle3D Research
https://rickee withhihhhhhhggh.b......... wwtggt.ittteaeeeeeeeeeee io/FashionR2//
Abstract
Modeling and producing lifelike clothed human images has attracted researchers' attention from different areas for decades, with the complexity from highly articu- lated and structured content. Rendering algorithms decompose and simulate the imaging process of a camera, while limited by the accuracy of modeled variables and the efficiency of computation. Generative models can produce impressively vivid human images, however still lacking in controllability and editability. This paper studies photorealism enhancement rendered images, leveraging generative power from diffusion models the controlled basis of rendering. We introduce a novel framework to translate rendered images into their realistic counterparts, which consists of two stages: Domain Knowledge Injection (DKI) and Realis- tic Image Generation (RIG). In DKI, we adopt positive (real) domain finetuning and negative (rendered) domain embedding inject knowledge into a pretrained Text-to-image (T2I) diffusion model. In RIG, we generate the realistic image corre- sponding to the input rendered image, with a peetturepresevving Attention Control (TAC) to preserve fine-grained clothing textures, exploiting the decoupled features encoded in the UNet structure. Additionally, we introduce SynFashion dataset, featuring high-quality digital clothing images with diverse textures. Extensive experimental results demonstrate the superiority and effectiveness of our method in rendered-to-rees image translation.
1 Introduction
Modeling and simulating digital humans and clothing has achieved significant progress 12 3 while leveraging these 3D assets for fashion e-commerce still remains a challenging problem. Due to the imperfection of 3D models and the approximation in rendering algorithms, rendered images cannot yet replace fashion photos taken by a camera, with deficiency in the realism of rendered human faces and skin, clothing shape and fabric, etc. This paper studies transferring rendered fashion images into their realistic counterparts, which is inherently an Paage-to-Image (I2I) translation problem.
5
Existing works on improving the realism of rendered images mainly resort to retrieving and blending real image patches (6), or train GAN-based network 17 [ 9 due lack of paired training data, Another line of works can tackle this problem general 121 translation [10]
[3] However,
these methods may still suffer from several limitations: Firstly, their image generation pipelines have limited power utilize real image resources for highly-detaile enhancement and may suffer from instability and mode collapse from adversarial training. Moreover, they either focus on indoor/outdoor
Work done during an internship at Style3D Research.
'Thee authors contributed equally to this work. Corresponning author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

scene enhancement while keeping coarse object-level semantic layout, or try to maintain face identity in training through loss constraints on sketches, and thus have difficulty in preserving fine- mne-grained texture in clothing images.
In this paper, we propose novel framework based on diffusion models for rendered-to-real fashion image translation to address above limitations. Our main idea consists of two aspects: Firstly, we propose to leverage abundant generative prior from pretrained Text-to-Image (T2I) diffusion models 14. and apply simple adaptation to realistic image generation under the guidance of distilled rendered prior. Secondly, we adopt texture- preserrvigg mechanism by extracting spatial image structure through attention from an inversion pipeline.
To achieve this, design a diffusion-based method consisting of two stages: Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG). During DKI, we first finetune a pretrained T21 diffusion model [14] on real fashion photos with derived captions from BLIP [15] adapt its capability in generating high-quality images to our target domain. After this adaptation, we propose to guide the image generation towards the negative direction of rendered effect. Inspired by Textual Inversion [16], we distill a general rendered "concept" with thousands of rendered fashion images by training a negative domain embedding vector based on the adapted base model. During RIG, we employ DDIM inversion [17] pipeline to first invert a rendered image into the latent noise map, and then generate its corresponding real image using the previous embedding as a negative guidance [18] Similar to recent training-frees free controls in T2I generation method [19] 20 21 [22]. we discover that the attention map in the shallow layers of the UNet contains rich spatial image structure and can be used for fine-grain fine-grained texture preuurrreesriinngggg during the generation. Specifically, we inject query and key of the self-attention from the rendered image inversion and generation pipeline to the rendered-to-res image generation pipeline. This largely improves the consistency of intricate clothing texture details.
We evaluate our method on public rendered Face Synthetics dataset [ and our collected SynFashion Dataset with fine-grained digital clothing and abundant texture variations. Empirical results comparing to previous works and experimental analysis demonstrate the efficacy of our method. Our main contributions are three-folds:
(1) We propose novel framework to address mandered-to-real fashion image translation by utilizing generative prior from pretrained diffusion models.
(2) We inject rendered-to-res domain knowledge into a pretrained T21 diffusion model through positive domain finetuning and negative domain embedding, and design a eexture-preserving attention control to preserve fine-gre ane-grained clothing textures during the translation.
(3) We collect a high-quality rendered fashion image dataset using the professional design software Style3D Studio, and plan to release the data with our paper to promote research in this important area.
2 Related Works
2.1 Rendered-to-real Image Translation
Improving the realism of rendered images has been long-standing problem due the inherent limitations of rendering pipelines and the rich potential for commercial applications. CG2Real [6] proposes to retrieve similar images from large collection of real photos and then applies local style transfer to upgrade color, tone and texture of the CG image. Deep CG2Real 7 adopts two-stage deep learning framework to first transfer OpenGl. images to PBR (Physically-Basee images, and then translates PBR to real images, disentangling lighting and texture in CycleGAN- like [23] framework. [8] enhances photorealism under the guidance of set of input G-buffers and learns the network with a perceptual discriminator. [9] proposes to learn rendered image generator for human faces, which can encode the same face identity but different "style" from a real face image generator, based on StyleGAN [24] 25). These methods all utilize limited data for generative training. while we propose to adapt diffusion models pretrained on large datasets for better image generation quality. Besides, applying these methods to fashion images often leads to the failure to preserve fine-grained clothing textures.
Rendering)
2.2 Image-to-image Translation
Transferring a rendered fashion image into its realistic counterpart is inherently an image-to-iman (I2I) translation problem, which has attracted wide interest in different realms of research 26] 27 28

---

Stage Domain Knowledge Injection
Staaee Realistic Image Generation
Dmages
Fine-tune
Rendered Image
Enecder
Zr
2
Z
Zo
TexTuie preserving Afterttion Control
Pre trained mode
do
Desoder
Realistic Image
Inversion
Demain Inversion
Dr
Zo
Neggttive Domain Embedding
Cross oftention
Bendened
Tuned modil
Figure I: The overall pipeline of our proposed method.
29 30
and applies pixel-wise regularization based on paired training data, which is unavailable in many problem settings. Cycle-GAN [23 proposes to utilize cycle consistency 85
Pix2Pix 32 utilizes conditional-Ganl
33 34
and optimizes a two-sided mapping between input source domain and output target domain. CUT 10 addresses the computational redundancy
36 37
and over-restriction this framework by simplifying it to one-sided [39] 40 [ and introduces
proposes an iterative refinement method based on Schrödinger bridge to overcome potential mode collapse in GAN generation, while still has difficulty in faithfully translating high-resolution images. Different from general 12I tasks and domain adaptation, our method focuses on photorealism enhancement and utilize more target-domain real photos for high-quality generation training, and thus can deal with imbalanced source-target training set. Style transfer [44 45 46) is a specific type of I2I task and manage to transfer input source image to arbitrary style [13
patch-wise contrastive loss 41
42 43
for refined local constraints. UNSB 12
47
48
49
50 given
one/few-shot target domain images as reference. These methods mainly focus on transferring style attributes like semantics, brushstrokes, colors, or material, while rendered-to- and enhancing complicated fine-grained details. Human/portrait relighting [51 (2) modifies the nuanced lighting condition in the input image, while does focus on enhancing realism and should leave geometry and materials untouched. super-resolution methods [53] 54 55 56 (57 (58) address
requires preserving
detail enhancement, while their success largely relies on synthesizing pseudo low-resolution images to obtain training pairs [59 60], which is non-trivial for rendered-to-ree problem.
2.3 Diffusion- based Image Synthesis
61 62 based on diffusion models 63 64 65] opens up new opportunity for advancing rendered to-reel image translation. Many works have explored the possibility of utilizing abundant generative prior in pretrained diffusion models. Some [66] [6] apply the adaptation of generation for new concept with few images, through either finetuning the base model [66], or optimizing a text embedding [16]. Others (67 68 leverage translation lies in the nuance
Recent progress in Text-to-Image (T2I) generation 14
text as guidance to edit given image. However, rendered-to of changes, which too subtle to define as a "concept" or to capture with a few images. [69 70] leverage diffusion models for texture estimation PBR synthesis, while mainly focusing on the generation certain variables for the rendering pipeline, rather than subtle modification preset variables in given input image. Additionally, [19 20, 21 22 discover that the attention in the SD UNet captures rich image features and can apply to content preservation and modification. In our work, utilize self-attention in shallow layers from the rendered image inversion, to impose the consistency fine- ane-grained texture in image translation.
Method
3.1 Preliminaries
Latent Diffusion Models. In diffusion framework, the forward diffusion process begins by generating noisy images 2, from clean images zo sampled from specified data distribution, accompanied by

---

their respective noise labels c. These pairs are used to train a score estimator 771 usually based on the UNet architecture. The score estimator can serve as an effective approximation of the score function V (ogp(()) which directs the inverse denoising process to generate new data samples.
With distinguished capabilities in synthesizing images, the Latent Diffusion Model (LDM) [14 is selected as the backbone of our method. The LDM employs pre-trained AutoEncoder to transform the diffusion process from pixel space to latent space and integrates a conditional branch, facilitating faster training and more flexible embedding of conditions. Specifically, the pre-trained Encoder (())irss encodes images into latent space E(x). Following this, the score estimator network cæ is trained by taking the latent 2, step and conditions as input to predict the noise labels:
min =0= ||e 6o (z,, 4, c)ll
0
For text to image generation task, condition is usually the text embedding generated from text prompt through a tokenizer and a pretrained CLIP 72 model c = r(y)- The intermediate noisy latent 2 is generated through the formula (64]
(2) ā is the cumulative product of the noise coefficients at each step. During the sampling process, the trained score estimator takes random Gaussian noise as input, along with text embedding as condition. It progressively predicts the noise added at each step, completing the denoising process to obtain åg. The final image obtained by the pretrained decoder ř. D(āg).
2e Va(()zo
(()e, (0,
Textual Inversion. Textual inversion [6 introduces a new paradigm T21 generation models, allowing the model to learn a new concept by setting a placeholder token "IC]" and obtaining the corresponding text embedding as a learnable vector. This vector is then trained and optimized using a few images represent this new concept:
i = arg (tn) || c0(2,, e)
(3)
During training, the network parameters are all fixed, only the embedding is optimized.
DDIM Sampling and Inversion. Inversion is an effective method for finding the corresponding noise map of an image and achieving training-free control during the generation process. DDIM inversion is widely used due its clear principles and easy implementation. The DDIM sampling process is 17
21
âees (Zt, c)
+ VI- 1-- (ze, t,
Vox,
By simply assuming 2t-1 R3 2t and rewriting the sampling process in reverse direction, the following DDIM Inversion [17 formula is given:
2t 2-1 VI 0;-1c0 (zt-1,t,c) VI arCo (2,1,,,c)
Va-1
Unlike direct noise addition, the DDIM Inversion allows for the original information of the image to be well preserved, enhancing the stability in the subsequent generation process.
3.2 Overall Pipeline
Given a computer-rendered fashion image Tegs the goal of our method is to transform it into a corresponding realistic image x, while preserving the garment's detailed textures. Defining realism and helping model understand what is "realistic" remains an open question. The challenge can be divided into two sub-tasks: one is making the fashion image appear realistic by enhancing aspects like wrinkles, lighting and color, which reflect true-to-life expressions. Another one is to maintain the texture details of the garment to achieve fine-grained, controllable generation.
As shown in Fig. M our method comprises two stages: Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG). During the DKI phase, we infuse the model with information from both the rendered and realistic domains through fine-tuning and domain inversion. the subsequent generation phase, we utilize negative domain embedding Una to stimulate the model's potential for generating realistic images and employ self-attention control to preserve texture details. For better understanding, details will be further elaborated in Section 3.3 and Section 3.4

---

3.3 Domain Knowledge Injection
Target Domain Knowledge Injection To enhance the ability of the base SD model to generate realistic images, especially concerning the appearance of garments and models, we use real studio- shot images Str to fine-tune the base model. This process injects real domain information into the model, thereby increasing its potential to generate authentic visual details, the fine-tuning process can be formulated as:
arg B (at ||c co(ze,t, etr)ll2
<a
where Eg is the pretrained SD model, Uty the embedding of the text description the Zty-
Source Domain Knowledge Injection For the source domain rendered data, we hope that the model can understand its characteristies and deviated from the rendered data manifold much as possible during the generation process. After the first step fine-tuning, we assume that the model has already enhanced its representation of the real domain manifold. If can make the model deviate from the rendered data manifold, it can better express the characteristics of realistic images. Inspired by the concepts of Textual In- version and Classifier-Frn Free Guidance (CFG) with negative prompts, we ex- pand the concept of Textual Inversion Domain Inversion. We train neg- ative domain embedding on a fine- tuned base model using a large num- ber of rendered images. This negative domain embedding guides the model to avoid certain content, here is the rendered domain characteristics, dur- ing the generation process.
2o v'd
Qui
Vis
>
s0 softman(
Vig
project
PiytK
Qui
K
V:
so coftmnan
Vv
project
vd
Q.
Kr
Specifically, given that textual descrip- tions of what is real and rendered are limited, it is difficult to guide the model to generate images with satisfactory realism or to precisely direct it not to produce images with rendered feel using text prompts only. Therefore, we consider using negative domain embed- ding Una trained on large number of rendered images for guidance to inject the rendered domain knowledge to the model. It's worth nothing that unlike textual inversion, which typically optimizes a small embedding space with few images to represent a specific concept, such as a particular object in personalized generation or an easily expressible style. The concept of rendered domain in our task is much more general. Using small embedding space corresponding to few images to represent this would easily lead to over-fitting to the content of the training images. We use the largest available embedding size to train the negative domain embedding, which is corresponding the placeholder token size 75:
Figure 2: The diagram of Texture Texture-preserving Attention Con-
trol (TAC).
Und arg atr.
||e (,,
During the training of negative domain embedding, we freeze the parameters in the fine-tuned model Eg, and find the Und through direct optimization with certain number of rendered images.
3.4 Realistic Image Generation
Negative Embedding Guidance After domain knowledge injection, we can use the negative domain embedding to guide the model in generating realistic images. During each denoising step, the negative domain embedding guidance is defined by:
Eg (Zty L, Vna) = w (ze, (,,,,,,))) Uo)) + (1 - w) eg (Zz, Un4)
where ee denotes the embedding of Null text. With guidance scale larger than 1, the negative domain embedding becomes effective. Unlike traditional CFG guidance, here we do not use any positive prompts processed through CLIP to obtain the embedding conditions. Instead, we directly
5

---

Source Image
CUT
SANTA
VCT
UNSB
Ours
Figure 3: Results on proposed SynFashion Dataset. (Please zoom in for details.)
employ a Null embedding. The initial noise latent is obtained through DDIM inversion of the given rendered image. During the denoising process, we replace the CLIP conditioning branch, since fine-tuned model, it can interact more effectively with
the negative domain embedding trained the base model's latent space. This consistency allows for more precise adjustments in the latent manifold compared embedding derived from text via CLIP.
presurr-peeeriing Attention Control (TAC) Inspired by previous work [19 22).
the attention
features in the diffusion UNet, which includes both cross attention and self attention, hold rich information critical for generating the new images. Cross attention typically handles the attributes and semantics of the generated image, while self-attention maps play crucial role in preserving geometric shapes and intricate details. The initial noise latent 2, derived from the DDIM inversion of the original rendered image can be used unconditional generation and extract the texture related attention features shown in Fig. 2 However, directly replacing all self-attention maps can lead to a decrease in the realism of the generated images. We argue that this because the attention map contains both the texture details of the garment and the general rendered domain characteristics. Therefore, we propose to control the self attention feature only in the shallow layers of the denoising UNet to decouple the texture details feature from the general rendered domain features. During the implementation, we also find that the deep feature spaces with higher downsampling rates, becomes challenging identify features related to the texture details. Thus, our TAC is defined as:
1Q K Qº K
if i F otherwise
- (Q'g- Koy Qr Kt,t)
=0
where y is the parameter that indicates how many steps before the TAC should be applied and is the feature size of different layers, only those layers exceeding the specified size F undergo TAC, particularly in the shallow layers. Specifically, the eg-domain self-attention features are derived from the reverse sampling process starting from the noisy latent, which is obtained by performing DDIM inversion on the input image latent. In contrast, the r-domain self-attention features differ due the incorporation of negative domain guidance and the self-attention injection.

---

Source Image
CUT
SANTA
VCT
UNSB
Ours
JUM
Figure 4: Results the Face Synthetics dataset. (Please zoom in for details.)
4 Experiments
4.1 Datasets
To evaluate our method and conduct comprehensive comparisons, we introduce a high-quality rendered fashion image dataset, named Synthetic Fashion (SynFashion), with the professional garment design software Style3D Studio. SynFashion consists of 10k rendered images in 20 categories, including pants, T-shirt, lingerie and swimwear, half skirt, hoodie, coat, jacket, set, home-wear, hat, Hanfu, jeans, shorts, down jacket, vest and camisole, shirt, suit, dress, sweater and trench coat, For each category, we Style3D Studio to build 10 to 40 projects in different 3D geometry with corresponding texture and design, and then randomly sample several new textures to change its appearance. There are overall 375 projects in 3D and 500 additional texture collected from Internet. For each textured 3D geometry, we render four views, including front, back, and two randomly sampled views. After rendering, we crop the enlarged garment area of each image and resize 768 x 1024. Due to legal issues, some of the images contain a digital human figure but not the complete face. To supplement the evaluation on rendered human faces, also conduct experiments on the public available Face Synthetics dataset [I] with its first 10k images.
4.2 Implementation Details
Implementation. We implement our method with pretrained Stable Diffusion (SD) model and finetune the base model with 2500 realistic images at 1024 x 1024 resolution for source domain knowledge injection. The finetuning uses images from iMaterialist (Fashion) 2019 FGVC dataset (73) based on the publicly available SD v1.5, and is conducted on 2 RTX 4090 with a batch size of Based on the finetuned model, we train our negative domain embedding with 2500 rendered images on a single RTX 4090 with batch size of 1. The rendered images are resized to the resolution of 512 512. The placeholder embedding size is 75 and the learning rate 5c-4. During sampling, we perform DDIM sampling with default 50 denoising steps with denoising strength of 0.3 as default. The y is set 0.9 as default, which means that the TAC is performed on the first 90% of sampling steps. Only the attention maps in the first and second shallow layers are used for TAC. Note that the denoising strength and y may changed to obtain different level of image translation. We compare our method with three state-of-the-arr unpaired image-to-image translation method, CUT, SANTA and UNSB, and one diffusion-based style transfer method VCT. For CUT, SANTA and UNSB, we train the models for about 400 epochs following the official code with same training data.
7

---

Table 1: Quantitative comparisons on Face Synthetics and SynFashion datasets.
Dataset
Face Synthetics
SynFashion
(ID](std)
(stP))(std)
(SIM1(std)
KIDU(std)
LPIPS_(std)
(SIM)(std)
CUT 10
80.553 (2.44 42.4
0.365 (0.073)
0.664 (0.079)
59.238 (1.599)
0.170 (0.060)
0.847 (0.067)
SANTA m
90.390 (2.929)
0.387 (0.079)
0.618 (0. 10.)
61.636 (1.628)
0.294 (0.067)
0.741 (0.082)
VCT 13
74.445 (2.273)
0.096 (0.027)
0.807 (0.072)
59,489 (1.499)
0.178 (0.058)
0.807 (0.085)
UNSB 12
76.389 (2.465)
0.229 (0.069)
0.818 (0.070)
59.496 (1.453)
0.130 (0.040)
0.891 (0.054)
Ours
73.871 (1.973)
0.121 (0.035) 0.831 (0.068)
54.720 (1.362)
0.067 (0.025)
0.881 (0.055)
Table 2: User studies on overall realism, image quality and consistency. The table shows the percentage of votes that existing methods preferred to ours.
Dataset
Face Synthetics
SynFashion Image Quality
Overall Realism
Image Quality
Consistency
Overall Realism
Consistency
CUT
0.529%
0.529%
13.175%
8.994%
6.878%
16.931%
SANTA
0.922%
1.383%
12.304%
3,333%
5,238%
11.571%
VCT
5.952%
14.286%
20.714%
2.041%
6.122%
18.367%
UNSB
4.511%
6.767%
21.278%
9.821%
9.821
26.607%
4.3 Results
Qualitative Results Fig. and Fig. show the visual comparison between our method, CUT [10] SANTA [1I UNSB 12 and VCT [3 the SynFashion and Face Synthetics datasets. As can be seen from the figures, both the CUT and SANTA methods exhibit some degree of image degradation and fail to effectively learn the concept of image realism from data across rendered and real domains, thus enable to generate realistic images. The diffusion based style transfer method VCT maintains image quality but fails to extract realistic image features from the guidance image, also resulting in the loss of image details. Compared to previous methods, the UNSB method achieves better consistency in terms of content, but like CUT and SANTA, it performs poorly in maintaining color fidelity and the realism effect is not good. The proposed method effectively enhances the overall realism of the image, particularly in capturing the facial and hand features of models, well as the texture and wrinkle details of the garment.
Quantitative Results The absence of ground truth for rendered-to- to-l translation and domain gap between the source rendered and target real domains make quantitative evaluation challenging.
Following the previous work R) we use KID to evaluate the realism of the generated images and the average SSIM and L.IPS to assess content similarity. For each dataset, we use the 7500 testing result images from each method and calculate the KID against the realistic images and the SSIM/LPIPS against the rendered images. As shown in Tab. [ our method shows significant improvements in terms realism as well as overall texture and content consistency. The standard deviations here show the variance over test inputs for fixed model to demonstrate the stability and generalization ability,
User Studies We adopt user studies to provide more quantitative insight into perceived realism, image quality, and consistency to input rendered images. We follow StyleDiffusion [74) style-transfer and compare our method to previous works in pairs. Specifically, we randomly sample 100 image pairs from each dataset for user evaluation. Each pair contains one image generated by our method and a corresponding image generated by another comparison method, presented side by side in random order. Users are asked to assess the images based on three criteria: 1) which result appears more realistic, 2) which result demonstrates overall better image quality, and 3) which result shows better consistency with the reference image.
We collected approximately 2,000 votes per question from 20 users and present the percentage of votes where existing methods were preferred over ours in the Tab. 2 Lower percentages indicate that our method was favored over the competitors. Our approach garnered strong preference in terms of overall realism and image quality, while also showing clear advantage in maintaining consistency with the reference images.
8

---

Source image
w/o DKI
target DKI
w/o
Ours
Figure 5: Visual examples of ablation study in drop-one-out manner. (DKI: Domain Knowledge
Injection. TAC: Texture- eexture-preeering Attention Control.)
Table 3: Ablation study in a drop-on-out manner.
Dataset
Face Synthetics
SynFashion
KID,(std)
(PIPS,(std)
(SIM?(std)
KIDI(std)
[.PIPS[(sul)
SSIM)(sid)
w/o souree DKI
77.376 (2.063)
0.107 (0.029)
0.857 (0.059)
58.520 (1.902)
0.059 (0.019)
0.903 (0.065)
w/o target DKI
78.927 (2.134)
0.114 (0.031)
0.845 (0.063)
60.186 (1.623)
0.064 (0.022)
0.897 (0.056)
w/o TAC
69.349 (1.485)
0.253 (0.070)
0.720 (0.085)
51.392 (1.083)
0.183 (0.047)
0.794 (0.074)
Ours
73.831 (1.973)
0.121 (0.035)
0.831 (0.068)
54.720 (1.362)
0.067 (0.025)
0.881 (0.055)
4.4
Ablation Study and Further Analysis
We conduct ablation study on two datasets in a drop-one-out manner and evaluate the performance of each module the proposed method during inference and analyze the impact on the final results. As shown in Fig. []] without source DKI (embedding), the fine-tuned base model tends to recover the input rendering image with DDIM inversion. Without target DKI (fine-tuning), the rendering effect slightly decreases but the output is still not real enough due to lack of concentrated knowledge on real human and clothing. Without TAC, the semantic structure such as face identity and clothing design can significantly deviate from the input. The quantitative results are in Tab. B
Fig. shows the trade-off between image realism and texture preservation. With high denoising strength, the generated images resemble realistic images more closely but retain fewer details from the original rendered image. Increasing the TAC ratio helps to better preserve the texture details and facial features. Unlike other content preservation techniques such as inpainting, which can lead to potential visual incoherence, our TAC seems blend the attention features smoothly into the generation process and cause no obvious coherence issues.
5 Conclusion
In this paper, we introduce a novel diffusion- based framework for rendered-to-real fashion image translation and create a high-quality rendered fashion image dataset (SynFashion), which includes 10k images with multiple classes. With Domain Knowledge Injection (DKI) and prxture-r Attention Control (TAC), our method can successfully translate the rendered fashion image into its realistic counterpart with significant realism improvement and texture details preservation. Extensive experimental results demonstrate the superiority and effectiveness of our method.

---

Source image
Target Image
%
Ratio of TAC control steps
90 %
Denolsing Strength
Figure 6: A visual example of tuning TAC ratio and denoising strength.
Table 4: Comparison of memory required and testing time across different methods.
CUT
SANTA
VCT
UNSB
Ours
Memory Required (GB)
3.3
4.5
22
7.4
7.7
Testing Time (s)
0.38
0.33
62.47
Limitations and social impacts While our method achieves superior results on this challenging task, there are still several problems to be further explored. In this work, simply use DDIM inversion to extract cexturerreltted attention features. However, the inversion process slows down the generation, requiring approximately one minute to translate image with resolution of 768 x 1024. This could potentially be accelerated by recent inversion-fis free methods. We test the inference time and resource consumption for 51 512x512 image which is also based on diffusion, our method takes much less memory and time during testing as we do not need to perform additional optimization for each testing image. Our method cannot handle real-time applications for now, but has potential for improvement with future integration with SD Turbo or SD Lightning. Additionally, for different images, finding the optimal balance between the TAC ratio and denoising strength may require more empirical refinements to achieve the best result, Due to limitations on computational resources, experiments were not conducted models such as SDXL 75 Given that our method is based on SD1.5 and for human- related content generation, potential negative societal impacts of exploiting this method could be violation of portrait rights, racial bias, or inappropriate content in generation when the denoising strength is high. Relative solutions can include but are not limited to using authorized, diverse and balanced training data and training detection models to prevent inappropriate content generation.
RTX 3090, shown in Tab. Note that comparing to VCT,
more advanced
6
Acknowledgments
This work supported part by the National Key Research and Development Program China (No: 2021 YFF0501503) and by the Talent Program of Zhejiang Province (No: 2021R51004).
10

---

References
[1] Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas Cashman, and Jamie Shotton. Fake till you make it: face analysis in the wild using synthetic data alone. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3681-3691, 2021.
[2] Astitva Srivastava, Pranav Manu, Amit Raj. Varun Jampani, and Avinash Sharma. Wordrobe: Text-guided generation of textured 3d garments. arXiv preprint arXiv:2403.17541 2024.
[3] Botao Wu, Zhendong Wang, and Huamin Wang. A gpu-based multilevel additive schwarz preconditioner for cloth and deformable body simulation. ACM Transactions on Graphics (TOG), 4144:1111, 2022.
[4] Zhendong Wang. Yin Yang, and Huamin Wang. Stable discrete bending by analytic eigensystem and adaptive orthotropic geometric stiffness, ACM Transactions Graphics (TOG), 42(66):--1. 2023.
[5] Yifei Li, Hsiao-yu Chen, Egor Larionov, Nikolaos Sarafianos, Wojciech Matusik, and Tuur Stuyck. Diffavatar: Simulation-ready garment optimization with differentiable simulation. arXiv preprint 1.Xiv:2311.12//99 2023.
[6] Micah K Johnson, Kevin Dale, Shai Avidan, Hanspeter Pfister, William T Freeman, and Wojeiech Matusik. Cg2real: Improving the realism of computer generated images using a large collection of photographs. IEEE Transactions Visualization and Computer Graphics, 1709):173-1225,, 2010.
[7] Sai Bi, Kalyan Sunkavalli, Federico Perazzi, Eli Shechtman, Vladimir G Kim, and Ravi Ramamoorthi. Deep cg2real: Synthetic-to-real translation via image disentanglement. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2730-2739, 2019.
[8] Stephan R Richter, Hassan Abu AlHaija, and Vladlen Koltun. Enhancing photorealism enhance- ment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4502):1700-1715. 2022.
[9] Luyuan Wang. Yiqian Wu, Yongliang Yang, Chen Liu, and Xiaogang Jin. Enhancing the authenticity of rendered portraits with identity-consistent transfer learning. arXiv preprint arXiv:2110.04194, 2023.
[10] Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Zhu. Contrastive learning for unpaired image-to-image translation. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 319-345. Springer, 2020.
[11] Shaoan Xie. Yanwu Xu, Mingming Gong, and Kun Zhang, Unpaired image-to-image translation with shortest path regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10177-10187, 2023.
[12] Beomsu Kim, Gihyun Kwon, Kwanyoung Kim, and Jong Chul Ye. Unpaired maage-to-image translation via neural schrödinger bridge. arXiv preprint arXiv:2305. 15086, 2023.
[13] Bin Cheng, Zuhao Liu, Yunbo Peng, and Yue Lin. General image-to-image translation with one- shot image guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22736-22746, 2023.
[14] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[15] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference machine learning, pages 12888-12900, PMLR, 2022.

---

[16] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2022.
[17] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprini 20Xi:22000002502.. 2020.
[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[19] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. houg-ad--layy diffusion features for text-driven maage-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1921-1930, 2023.
[20] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen -rr Prompt-to-promp image editing with cross attention control. arXiv preprint arXiv:2208.011: 016666 2022.
[21] Mingdeng Cao. Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22560-
22570, 2023.
[22] Bingyan Liu, Chengyu Wang. Tingfeng Cao, Kui Jía, and Jun Huang. Towards understanding cross and self-attention in stable diffusion for text-guided image editing. arXiv preprint arXiv:2403.00 03331,, 2024.
[23] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[24] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110-8119, 2020.
[25] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IÉEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[26] Tao Chen, Ming-Ming Cheng, Ping Tan, Ariel Shamir, and Shi-Min Hu. Sketch2photo: Internet image montage. ACM transactions on graphics (TOG), 28(5):1-10, 2009.
[27] Aaron Hertzmann, Charles Jacobs, Nuria Oliver, Brian Curless, and David H Salesin. Image analogies. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 557-570. 2023.
[28] Lara Raad and Bruno Galerne. Efros and freeman image quilting algorithm for texture synthesis. Image Processing On Line, 7:1-22, 2017.
[29] Yichang Shih, Sylvain Paris, Frédo Durand, and William T Freeman. Data-driven hallucination of different times of day from single outdoor photo. ACM Transactions on Graphics (TOG),
32(6):1-11, 2013.
[30] Ting- Mhng-Chun Wang, Ming- Yu Liu, Jun Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 8798-8807, 2018.
[31] Elad Richardson, Yuval Alaluf, Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 2287-2296, 2021.
12

---

[32] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-ing to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 1125-11344 2017.
[33] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[34] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 1411. 1.111,1844 1784, 2014.
[35] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances neural information processing systems, 30, 2017.
[36] Tacksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In International conference on machine learning, pages 8857-1865. PMLR, 2017.
[37] Zili Yi, Hao Zhang. Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision, pages 2859,2857, 2017.
[38] Sihan Xu, Ziqiao Ma, Yidong Huang, Honglak Lee, and Joyce Chai. Cyclenet: Rethinking cycle consistency text-guided diffusion for image manipulation. Advances in Neural Information Processing Systems, 36, 2024.
[39] Huan Fu, Mingming Gong, Chaohui Wang. Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. Geometry-consistee generative adversarial networks for one-sided unsupervised do- main mapping. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[40] Sagie Benaim and Lior Wolf. One-sided unsupervised domain mapping. Advances in neural information processing systems, 30, 2017.
[41] Chanyong Jung, Gihyun Kwon, and Jong Chul Ye. Exploring patch-wis wase semantic relation for contrastive learning in image-to-image translation tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18260 18269, 2022.
[42] Weilun Wang, Wengang Zhou, Jianmin Bao, Dong Chen, and Houqiang Li. Instance-wise hard negative example generation for contrastive learning in unpaired image-to-image translation. In Proceedings of the IEEC/CVF international conference on computer vision, pages 14020-14029, 2021.
[43] Chuanxia Zheng. Tat-Jen Cham, and Jianfei Cai. The ceatialll--orrelann loss for various image translation tasks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16407-16417, 2021.
[44] Bin Wang, Wenping Wang, Huaiping Yang, and Jiaguang Sun. Efficient example-based painting and synthesis of 2d directional texture. IEEE Transactions Visualization and Computer Graphics, 10(3):266-2) 277, 2004.
[45] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 2414-2422 2016.
[46] Leon A Gatys, Alexander S Ecker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. Controlling perceptual factors in neural style transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3985-3993, 2017.
[47] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. Advances neural information processing systems, 30, 2017.
13

---

[48] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
[49] Yingying Deng, Fan Tang, Weiming Dong, Wen Sun, Feiyue Huang, and Changsheng Xu. Arbi- trary style transfer via multi-adaptation network. Proceedings of the 28th ACM international conference multimedia, pages 2719-2721 2020.
[50] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. Inversion- style transfer with diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10146-10156, 2023.
[51] Manuel Lagunas, Xin Sun, Jimei Yang. Ruben Villegas, Jianming Zhang, Zhixin Shu, Belen Masia, and Diego Gutierrez. Single-image full-body human relighting. In Eurographics Symposium Rendering (EGSR). The Eurographics Association, 2021.
[52] Yu- YinYing Ych, Koki Nagano. Sameh Khamis, Jan Kautz, Ming- Yu Liu, and Ting-Chun Wang. Learning to relight portrait images via a virtual light stage and synthetic-to-rea adaptation. ACM Transactions Graphics (TOG), 41(6):1-21, 2022.
[53] Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang, Seesr: Towards semantics- real-world image super-resolution. arXiv preprint arXiv:2311.16518, 2023.
[54] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang. Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. arXiv preprint ANNv:22001116227 2024.
[55] Jianyi Wang. Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Ex- ploiting diffusion prior for real-world image super-resolution. arXiv preprint arXiv:2305.070/5, 2023.
[56] Haoyu Chen, Wenbo Li, Jinjin Gu, Jingjing Ren, Haoze Sun, Xueyi Zou, Zhensong Zhang, Youliang Yan, and Lei Zhu. Low-res leads the way: Improving generalization for super- resolution by self-supervised learning, arXiv preprint arXiv:2403.022 02601, 2024.
[57] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement, IFEE transactions on pattern analysis and machine intelligence, 45(4):4713-4726, 2022.
[58] Jingyun Liang, Jiezhang Cao. Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEF/CVF international conference on computer vision, pages 1833-1844, 2021.
[59] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1905-1914, 2021.
[60] Kai Zhang, Jingyun Liang. Luc Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4791-4800, 2021.
[61] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 1(2):3 2022.
[62] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 0..36479-36494 2022.
[63] Jascha Sohl- Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
14

---

[64] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[65] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[66] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[67] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[68] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[69] Kim Youwang. Tac-Hyun Oh, and Gerard Pons-Mon Moll. Paint-it: Text-to-texture synthesis via deep convolutional texture map optimization and physically- rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4347 4356, 2024.
[70] Dan Casas and Mare Comino-Trinidad SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image. In British Machine Vision Conference (BMVC), 2023.
[71] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1, 13456, 2020.
[72] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 1148-8763.. PMLR, 2021.
[73] David Shi, Maggie, Menglin Jia, Mihail Sirotenko, and Will Cukierski. imaterialist (fashion) 2019 at fgvc6, 2019.
[74] Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7677-7689, 2023.
[75] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 207012230. 2023.
15

---

A Appendix supplemental material
Rasterization
Ray Tracing
W TUSST
WHYY SE
Mctic Ee
WHNT YOU SE
PACIFI
HN MI CA
Lanis HATOUUGT
Rendered
PANTIUUGEE OUIGET
E
HATIQUGET
HATIOUGET
WE BRI
WI WARFYCHSE
WINTSOUSE YOGGG
CHICC
POOOG
Ours
BETTS HATIQUGT
HATOUUET
HATIOUGET
HATIOUGET
Figure Results on textual textures and different rendering inputs.
A.1
More implementation details
More details of our RIG algorithm is shown in Algorithm. I
Algorithm 1 Realistic Image Generation
Inputs:
2: Teg t- source rendered image
3: End <- negative domain embedding
4: 7, F < step threshold, feature size threshold
5: Algorithm:
6: E(Xeg)
7: år <- DDIM-inv(zo)
8: 27 2T // starting from same seed
9: for = to do
10:
21-1, Q' Kt DDDM-DMMM(pP))
11:
i < & f> then
12:
2 t Eg (8 vna) (Q) Qigi K3 +- }
13:
else
14:
2 x 2g (2t, Una)
15:
end if
end for
17:
Output: H, f- D(āu)
16

---

Table 5; Number of images in different categories of SynFashion.
Category
Pants
T-shirt
Lingerie Swimwear
Half Skirt
Hoodie
Coat
Jacket
Numbers
864
416
440
392
448
604
812
Category
Sct
Home-
Hat
Hanfu
Jeans
Shorts
Down Jacket
wear
Numbers
420
336
308
472
180
420
508
Category
Vest & Camisole
Shirt
Suit
Dress
Sweater
Trench
Coat
Numbers
388
476
1056
416
A.2 Results textual textures and different rendering inputs
As for rendering baselines, we build the 3D projects with Style3D Studio and use its integrated rendering tool based rasterization. Using UES could potentially improve the rendering quality but will not diminish the effectiveness of our method. To verify this, we use more advanced rendering techniques via ray tracing (based V-ray) to obtain rendered images, and method consistently demonstrates its advantages in realism. Two visual examples are shown Fig.
A.3 More results of realistic image translation
To further verify the performance of the proposed method in realistic translation tasks, additional experiments were conducted using the collected SynFashion dataset and the Face Synthetics dataset. The results are illustrated in Figure for Face Synthetics and Figure 9 for SynFashion.
A.4 More details of collected SynFashion dataset
provide detailed visualizations of the SynFashion dataset. The first column in each figure presents the front view of designed 3D garment object. Various texture patterns are assigned to each garment object, and the subsequent columns show the images with four different views. The number of images in each category is shown in Table. 5
Figure TO Figure 11
Figure 12 and Figure T3
17

---

Source Image
CUT
SANTA
VCT
UNSB
Ours
CLAA
MII
AUERI
TOARI
CUUBI
Figure 8: More comparison results on Face Synthetics dataset.
18

---

Source Image
CUT
SANTA
VCT
UNSB
Ours
Figure 9: More comparison results on SynFashion dataset.
19

---

Pants
T-shirt
Lingerle and Swimwear
Half Skirt
Hoodie
Figure 10: Examples of collected SynFashion dataset (Part 1).
20

---

Coat
Jacket
Set
Home-wear
Hat
Figure 11: Examples of collected SynFashion dataset (Part
21

---

Hanfu
Jeans
Shorts
Down Jacket
Vest and camisole
Figure 12: Examples of collected SynFashion dataset (Part
22

---

Shirt
Suit
Dress
Sweater
Trench Coat
Figure 13: Examples of collected SynFashion dataset (Part 4).
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the papaer's contributions and scope. Details can found at the end of Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work listed in Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper is more experimental and does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results in Section 4.1 (Datasets) and Section 4.2 (Implementation Details).
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

reproducible, they Answer: [Yes]
Justification: Data and code are released in the supplemental materials to preserve anonymity and will be made public available after review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//ipps.c//pil wttpccpppppppppeeeeeeeeeeeeeeeee
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and testing details shown in Section 4.2 (Implementation Details)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We follow evaluation metrics from previous works and use public evaluation code to compute quantitative results in our paper. Since quantifying realism is still an open question and the metrics not strictly aligned with the objective, which makes further statistical analysis a bit less informative than other well-defined problems, we only report the standard deviations to show the variance test inputs for fixed model and demonstrate the overall stability and generalization ability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The detailed information on the computer resources needed are listed in Section
4.2 (Implementation Details).
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https:
Answer: [Yes]
Justification:
This research conforms, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader social impacts are listed in Section 5 (Conclusion)
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
The only possible misuse of our work would be using high denoising strength generate potentially inappropriate images, which is warned in conclusion and will be mentioned when we release the model.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The existing assets are properly credited and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new dataset introduced in the paper is well documented in an anatomized link with the license of CCBY-NC-ND4.0.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does involve crowdsourcing nor reasearch with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "sykEcdSY12", "reviewer": "Reviewer_m8pA", "summary": "This paper proposed a modified framework for generating realistic fashion photos. The framework enhances existing diffusion models through a knowledge injunction pre-training process and an attention control mechanism during the generation process. Additionally, the author introduced a new dataset including high-resolution textures patterns of digital clothes. This dataset, combined with improvements in texture identification, enables the new framework to perform better compared to other frameworks.", "questions": "1. There is a problem regarding the diffusion models that where generated images exhibit significant randomness given the same segment mask. How does the author’s work address this problem?\n2. There is a confusion that in the introduction section, for DKI pre-training, the author states they applied a text-to-image diffusion model. However in the rest sections of the paper, it seems the pre-trained model is trained by images. How to understand this difference?\n3. Does this framework perform differently on various patterns of texture? For example, what is the outcome when a textual texture, such as a sentence, is given?", "limitations": "The authors have addressed some limitations of their framework, however it would be more comprehensive if they can add some in-depth discussion on potential negative societal impacts and provide relative solutions.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The author created a high-resolution dataset with more than ten thousand digital cloth texture patterns, which fills a gap in relevant fields.\n2. Compared to other frameworks, this framework had a better performance in terms of clothing texture and facial naturalness for given source image. It is an improvement in application of AIGC in everyday life.\n3. The framework impressively achieved transforming a clearly unrealistic image containing human faces into a plausible and realistic one.", "weaknesses": "1. There is no updating on diffusion structure but relies on pre-training and post-processing.\n2. The approach is more engineering application than academic paper.\n3. When translating images containing human faces and meaningful backgrounds, the framework seems to lose or even change the background.\n4. The structure and grammar of the paper could be further improved for better clarity and readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions in the proposed dataset and \"impressive\" performance of our framework, as well as the valuable feedback and suggestions. Below we address the concerns and questions in detail:\n\n> There is no updating on diffusion structure but relies on pre-training and post-processing.\n\nThe main contribution of our proposed method is in regard to novel domain knowledge tranfering with established T2I diffusion-based models, combining with specific content control through attention injection. The TAC module is an exploration on diffusion structure and a slight modification of the structure to achieve fine-grained texture preservation. Our training and inference process is standard but convenient, without needing additional optimization or hand-crafted post-processing during testing.\n\n> The approach is more engineering application than academic paper.\n\nAs discussed in general response, our paper studies an important scientific problem, brings out a new perspective, proposes a novel pipeline, establishes a comprehensive benchmark, asks some essential questions, has potential in introducing more resources, and would work as an inspiration for future work.\n\n> The framework seems to lose or even change the background.\n\nChanging image content is a well-known side-effect of image-to-image translation with generative models, especially for diffusion models.\n\nIn our paper, we propose a TAC module to mitigate this problem through attention injection from specific layers of the diffusion UNet. In particular, the high-level semantic structure such as face identity and clothing structure, as well as low-level texture patterns are largely preserved. \n\nThe background can consist semantic layout structure and texture details, which would lie in similar distribution in the image space and thus be preserved by our method, as shown in Figure 4. \n\nIn Figure 6, the background changes significantly when increasing denoising strength and decreasing ratio of TAC control steps, which is to show an effective range of the hyper-parameters and the trade-off between content preservation and realism enhancement.\n\n> The structure and grammar of the paper could be further improved for better clarity and readability.\n\nWe appreciate the suggestion and will modify our paper to improve its clarity and readability.\n\n> There is a problem regarding the diffusion models that where generated images exhibit significant randomness given the same segment mask. How does the author’s work address this problem?\n\nGiven the same input, the randomness can decrease when the denoising strength is lower and less noise is added to the input. However, to leverage the generative power for image translation, the denoising strength cannot be too low, as shown in Figure 4. In our work, we propose a TAC module to help preserve fine-grained structural image content, which can decrease its related randomness while better exploiting the diffusion model's generative power in other layers. Empirically, we found a combination of denoising strength 0.3 and TAC ratio 0.9 achieves the best balance.\n\n> There is a confusion in model training.\n\nIn Introduction Line 41, we mention that the T2I diffusion model is trained on real photos with derived captions from BLIP. In Method Line 166, we note that the text embedding is also an input in model training. We will further clarify this in our paper writing and Figure 1.\n\n> Does this framework perform differently on various patterns of texture? Such as a sentence?\n\nOur framework can robustly handle various patterns including textual texture. The second row in Figure 3, the last row in Figure 8, and two additional examples in Figure 1 of the attached PDF are some visual examples.\n\n> Add some in-depth discussion on potential negative societal impacts and provide relative solutions.\n\nWe thank the reviewer for suggesting this and will add more discussion on potential negative societal impacts, including potential violation of portrait rights, racial bias in generation, NSFW content, etc. Relative solutions can include but are not limited to using authorized, diverse and balanced training data and training detection models to prevent inappropriate content generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fIlS5pvii7", "reviewer": "Reviewer_SbYx", "summary": "The paper presents an advancement in the field of rendered-to-real image translation, particularly for fashion images. Its novel approach, focus on texture preservation, and contribution of a new dataset make it a valuable addition to the literature.", "questions": "Have you conducted any user studies to assess the perceived realism and quality of the generated images compared to existing methods?\n\nCan you discuss the trade-offs involved in maintaining texture details versus achieving overall visual coherence in the generated images?", "limitations": "The two-stage process involving Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG) can be computationally intensive. Training and deploying the framework may require substantial computational resources, which could limit its applicability in real-time or resource-constrained environments.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The Texture-preserving Attention Control (TAC) mechanism effectively maintains fine-grained details in clothing textures, which is crucial for realistic image generation.\n\nIntroduction of the SynFashion dataset fills a gap in the availability of high-quality rendered fashion images, facilitating further research in this area.\n\nExtensive experimental results demonstrate the superiority of the proposed method over existing techniques, highlighting its effectiveness in achieving realistic image translation.", "weaknesses": "The paper acknowledges that defining and quantifying \"realism\" remains an open question. More robust metrics or user studies could provide better insights into the perceived realism of the generated images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive comment and acknowledgement on the TAC design of our method, the value of our SynFashion dataset, and the experimental results. The suggestions on user studies and further discussion are very helpful. Below we address them in detail:\n> The paper acknowledges that defining and quantifying \"realism\" remains an open question. More robust metrics or user studies could provide better insights into the perceived realism of the generated images.\n\nDefining and quantifying \"realism\" is indeed an open question, given the nuanced change between the input rendered image and its realistic counterpart, which is unknown and non-unique. We compute SSIM and LPIPS (as suggested by Reviewer cE2m, please refer to the related rebuttal) to roughly evalute texture preservation to rendered images, and use KID to measure the distance between the outputs and real photos in distribution. However, these metrics still cannot accurately reflect the objective of translation in photorealism.\n\nWe thank the reviewer for suggesting user studies to provide better insights quantitatively, regarding the perceived realism, image quality, and consistency to input rendered images. We follow [1] StyleDiffusion (ICCV 2023) in style-transfer and compare our method to previous works in pairs. Specifically, we sample 100 image pairs from each dataset for a user to choose, where each pair consists of one image generated from our method and a corresponding image generated with anthor random method side by side in random order. The users are asked to choose (1) which result is more realistic and (2) which result has overall better image quality (3) which result has better consistency with the rendered image.  \n\nWe obtain approximate 2000 votes for each question from 20 users and show the percentage of votes that existing methods are prefered to ours in the following Table. The lower numbers indicate our method is more prefered than the competitors. Our method achieves overwhelming preference in overal realism and image quality, as well as obvious advantage in consistency.\n\n| FaceSynthetic Dataset | CUT     | SANTA   | VCT     | UNSB    |\n|-----------------------|---------|---------|---------|---------|\n| Overall realism       | 0.529%  | 0.922%  | 5.952%  | 4.511%  |\n| Image quality         | 0.529%  | 1.383%  | 14.286% | 6.767%  |\n| Consistency           | 13.175% | 12.304% | 20.714% | 21.278% |\n\n| SynFashion dataset    | CUT     | SANTA   | VCT     | UNSB    |\n|-----------------------|---------|---------|---------|---------|\n| Overall realism       | 8.994%  | 3.333%  | 2.041%  | 9.821%  |\n| Image quality         | 6.878%  | 5.238%  | 6.122%  | 9.821%  |\n| Consistency           | 16.931% | 11.571% | 18.367% | 26.607% |\n\n> Can you discuss the trade-offs involved in maintaining texture details versus achieving overall visual coherence in the generated images?\n\nMaintaining centain parts of an image during generation, such as texture details, can lead to potential visual incoherence. One example is inpainting with diffusion models. However, in our work, our proposed TAC tackle this issue resorting to attention injection from a dual diffusion branch. Emperically, we discover the shallow layers of diffusion UNet preserves texture details while deep layers are more related to semantic information, and the injection of self-attention from shallow layers can blend smoothly with other features in the generation branch. In Figure 6, we show the effect of different ratio of TAC control steps during diffusion generation. We can see that the coherence in the generated image is not an obvious issue in our framework.\n\n> The two-stage process involving Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG) can be computationally intensive. Training and deploying the framework may require substantial computational resources, which could limit its applicability in real-time or resource-constrained environments.\n\nComputational cost and resource requirements are indeed relevant issues of our method, and probably similar to many diffusion-based methods. We thank the reviewer for mentioning this and will add it into our discussion on limitation and future work. Besides, we conduct some analysis on computational cost comparing to other SOTAs. We test the inference time and resource consumption for a 512x512 image on an RTX 3090, as shown in the table below. Note that comparing to VCT, which is also based on diffusion, our method takes much less memory and time during testing as we do not need to perform additional optimization for each testing image.\n\n|          | CUT  | SANTA | VCT   | UNSB  | Ours  |\n|----------|------|-------|-------|-------|-------|\n| Memory Required / GB | 3.3 GB | 4.5 GB | 22 GB | 7.4 GB | 7.7 GB |\n| Testing Time / s | 0.38 s | 0.33 s | 62.47 s | 0.53 s | 7.98 s |\n\n[1] Wang, Zhizhong, Lei Zhao, and Wei Xing. \"Stylediffusion: Controllable disentangled style transfer via diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWLbs8VWBb", "reviewer": "Reviewer_cE2m", "summary": "This paper proposes a method to translate synthetic renderings of fashion shots into photorealistic images. It is based on Stable Diffusion; SD is first fine-tuned on the target domain; then a negative guidance token for the rendered domain is learnt; then the image is translated by DDIM-inversion in original SD followed by reconstruction with the fine-tuned SD under the learnt negative guidance (i.e. encoraging the image to look 'less synthetic').\nResults are presented on two datasets, FaceSynthetics and SynFashion; the latter is proposed in the present work. Quantitative and qualitative results show some improvement over four baselines.", "questions": "- The description of TAC (L211-226) is somewhat unclear; in particular it should be stated up-front where the two sets of self attention features are coming from. Are the CG-domain features from the DDIM inversion pass on the input image? And the R-domain those in the reverse process, which differ due to the negative guidance?\n\n- What are the standard deviations on the quantitative results?\n\n- What are the quantitative results from the ablations, including removing source & target DKI separately?\n\n- Figure 6 is not in fact an ablation experiment – it doesn't remove any component of the model to demonstrate importance; it merely shows the effect of varying certain hyperparameters. Please adjust the text/caption accordingly\n\n- Table 2 in App. A3 would be much easier to read if it were transposed. Also it is wrongly named in the text as \"Table A3\"\n\n- Reference to Table 3 at L279 is also broken", "limitations": "There is adequate discussion of limitations, and a very brief discussion of broader impacts.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The proposed pipeline is novel. The individual components are fairly standard (DDIM inversion; fine-tuning; dreambooth-style token optimisation), but combined in a sensible (and clearly motivated) way to solve the relevant task.\n\n- Curated qualitative results presented in the main paper are significantly than four baselines (CUT, SANTA, VCT, UNSB), with good adherence to textural details in garments but significantly more realism than the source renderings\n\n- Quantitative metrics on two synthetic datasets (FaceSynthetics and the proposed SynFashion) are better than the four baselines. In particular SSIM is higher (meaning details from the rendered images are preserved) while KID is lower (meaning the distribution of generated images is more realistic)\n\n- The paper is generally well-written, well-structured, and clear", "weaknesses": "- The paper feels rather insubstantial in terms of scientific contribution. It is not very surprising that the presented techniques can be combined into an effective pipeline, and there are not many insights for the reader. While it appears to be a sound piece of engineering, interest for the NeurIPS community is probably not so great.\n\n- Compounding the above point, the ablation study combines together under 'DKI' the two separate ideas of fine-tuning on the target domain, and learning a guidance signal 'away' from the source domain; this makes it unclear what their relative benefits are. Moreover, there are no quantitative ablation results, only two visual examples\n\n- Only SSIM and KID are given as metrics – why no other reconstruction metric (e.g. LPIPS, which is better correlated with human perception), and why not FID (which is still much more widely employed than KID)?\n\n- Measures of spread / statistical significance are not present, despite authors answering 'yes' to the relevant checklist question. The lack of variance measures in the quantitative results make it impossible to draw comparisons from these – particularly for SSIM where the difference between proposed method and best baseline is exceedingly small. For KID, 7500 images is on the low side for robustness of the metric.\n\n- The method appears somewhat 'conservative', in the sense that it favors preserving appearance over enforcing realism. See for example the outputs in fig. 8, where the model makes only minimal changes to the renderings, failing to reach photorealism for most of the rows", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the recognition and encouragement on our novel pipeline, significantly more real qualitative results, effective quantitative results, and generally well-written paper. We would also like to thank the reviewer for the rigorous analysis and suggestions to help us improve our paper. Below we address the concerns and questions in detail:\n\n> Scientific contribution, insights, and interests for the community\n\nWe believe this is a misunderstanding and maybe due to somewhat lack of elaboration in our paper. In general response, we further explain our scientific contributions. We do not intend to compose an engineering pipeline to somehow manipulate diffusion generation, but propose to study a fundamental research problem, which might not be so obviously achieved and could be interesting to the NeurIPS community.\n\n>  Quantitative results from the ablations\n\nWe conduct ablation study on two datasets in a drop-one-out manner. The quantitative results are in the table below and two visual examples are shown in Figure 2 of the attached PDF. \n\n| FaceSynthetic Dataset| w/o source DKI | w/o target DKI | w/o TAC| ours|\n|------------------------------------------|----------------|----------------|-----------------------------------|----------------|\n| SSIM (std)| 0.8572 (0.0592)| 0.8447 (0.0634)| 0.7199 (0.0849)| 0.8314 (0.0679)|\n| LPIPS (std)| 0.1072 (0.0285)| 0.1137 (0.0313)| 0.2529 (0.0703)| 0.1206 (0.0349)|\n| KID (std)| 77.376 (2.063) | 78.927 (2.134) | 69.349 (1.485)| 73.831 (1.973) |\n\n| SynFashion Dataset | w/o source DKI | w/o target DKI | w/o TAC| ours|\n|------------------------------------------|----------------|----------------|-----------------------------------|----------------|\n| SSIM (std) | 0.9033 (0.0654)| 0.8971 (0.0556)| 0.7937 (0.0741)| 0.8809 (0.0553)|\n| LPIPS (std)| 0.0587 (0.0194)| 0.0635 (0.0217)| 0.1827 (0.0472)| 0.0671 (0.0251)|\n| KID (std)| 58.520 (1.902) | 60.186 (1.623) | 51.392 (1.083) | 54.720 (1.362) |\n\nWithout source DKI (embedding), the model tends to recover the input rendered image with DDIM inversion noise.\n\nWithout target DKI (finetuning), the rendering effect slightly decreases but the output is still not real enough due to lack of concentrated knowledge on real human and clothing. Lacking either component in DKI will make the output closer to the rendered input (higher SSIM and lower LPIPS), but much less real (higher KID).\n\nWithout TAC, the semantic structure such as face identity and clothing design can significantly deviate from the input. Although the output is clearly real (lower KID), it is almost completely different from the intended human and clothing (lower SSIM and higher LPIPS).\n\n>  Other reconstruction metric (e.g. LPIPS, FID), standard deviations\n\nAs discussed in our paper and agreed upon by Reviewer SbYx, defining and quantifying \"realism\" still remains an open question. SSIM and KID roughly reflect the objective in enhancing photorealism while maintaining consistency. We follow the reviewer's suggestion and add LPIPS to evaluate texture preservation regarding rendered images, which is closer to human perception. The results are shown in the table below. We follow [2] and adopt KID rather than FID, for KID's unbiased nature and less sensitivity to sample sizes. Note that a good SSIM or LPIPS score does not guarantee better photorealism if the KID score is worse.\n\nWe' d also like to thank the reviewer for pointing out the missing information in statistical significance and add the standard deviations of the results in the table. The relatively low stds demonstrate the robustness of our method.\n\nAdditionally, as suggested by Reviewer SbYx, we conduct user studies to augment the quantitative evaluation. Our results in \"real human perception\" turn out be overwhelmingly better than other methods. Please refer to related rebuttal for more details.\n\n| FaceSynthetic Dataset | CUT| SANTA | VCT| UNSB| Ours|\n|------------------------|----------------|----------------|----------------|----------------|----------------|\n| SSIM (std) | 0.6635 (0.0786)| 0.6176 (0.1039)| 0.8072 (0.0723)| 0.8176 (0.0696)| 0.8314 (0.0679)|\n| LPIPS (std)| 0.3649 (0.0725)| 0.3872 (0.0788)| 0.0957 (0.0270)| 0.2287 (0.0685)| 0.1206 (0.0349)|\n| KID (std) | 80.553 (2.447) | 90.390 (2.929) | 74.445 (2.273) | 76.389 (2.465) | 73.831 (1.973) |\n\n| SynFashion Dataset| CUT| SANTA| VCT| UNSB| Ours|\n|------------------------|----------------|----------------|----------------|----------------|----------------|\n| SSIM (std)   | 0.8467 (0.0670)| 0.7408 (0.0824)| 0.8070 (0.0846)| 0.8909 (0.0542)| 0.8809 (0.0553)|\n| LPIPS (std) | 0.1699 (0.0603)| 0.2940 (0.0669)| 0.1783 (0.0577)| 0.1299 (0.0403)| 0.0671 (0.0251)|\n| KID (std)| 59.238 (1.599) | 61.636 (1.628) | 59.489 (1.499) | 59.496 (1.453) | 54.720 (1.362) |\n\n> Preserving appearance or enforcing realism.\n\nThe results we present in the paper do emphasize more on preserving appearance, under the assumption that the human model identity and clothing design should remain largely consistent to the input. However, the realism enhancement of our method could be more 'aggressive' given more tolerance to modifications to the input content. In Figure 6, we show an example of the trade-off in using different hyper-parameters.\n\n> The description of TAC is somewhat unclear.\n\nWe will modify the description of TAC.  The CG-domain self-attention features are derived from the reverse sampling process starting from the noisy latent, which is obtained by performing DDIM inversion on the input image latent. In contrast, the R-domain self-attention features differ due to the incorporation of negative domain guidance and the self-attention injection.\n\n> Figure, table captions and typos\n\nWe thank the reviewer for pointing out these issues and will carefully modify our paper.\n\n[2] Richter, Stephan R et al. \"Enhancing photorealism enhancement.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 45.2 (2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W18X0afUnp", "reviewer": "Reviewer_L3jv", "summary": "This paper proposes a diffusion-based method for enhancing the realism of fashion images generated through computer graphics pipelines. To this end, the paper introduces a Texture Preserving Attention Control for improving the appearance of the textures in garments in the generated images, and a domain-knowledge injection for improving the realism of the images. The paper presents SynFashion, a synthetic dataset for research on this topic, and provides an open-source implementation. Qualitative and quantitative analyses are provided.", "questions": "- Are there any applications for which the proposed modules could be used, that are not specific to garment generation?\n\nMissing references: \n- SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image (BMVC23)\n- Single-image Full-body Human Relighting (EGSR21)\n- Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation (TOG22)\n- Paint-it: Text-to-texture synthesis via deep convolutional texture map optimization and physically-based rendering (CVPR24)", "limitations": "The paper is up-front about its limitations. I appreciate this part of the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper tackles an important problem, which is that of enhancing the realism of images generated with computer graphics applications.\n- This paper presents a new dataset for research on this topic. The dataset looks very interesting, comprising a diverse set of textures and garments. I believe this dataset is a salient contribution of this work.\n- The proposed methods are sound, well motivated in the context of the application, and well presented.\n- Supplementary material provides additional insights on the results of these methods.\n- Code is provided which should enhance reproducibility. \n- Evaluation provides qualitative and quantitative results against some baselines, showing improved realism and preservation of the texture in the garments.", "weaknesses": "- Some of the contributions of this work are very tied to a specific application, which is that of generating images of textured garments. While the methods are sound, I believe this somewhat narrow application strongly limits the scope of this work. It is unclear how the proposed contributions, particularly the Texture-Preserving Attention Control could be leveraged for other applications.\n- The analysis of the related work is somewhat lacking. Please see \"Questions\" for a few suggestions.\n- The paper is sometimes lacking in clarity, being overly convoluted in its writing.\n- The proposed method is very computationally costly. While this is acknowledged by the authors, I think it is a significant limitation of this work. In this regard, the long computational times of this method make it unsuitable for many applications, including interactive virtual try-on. Further, I believe that a more powerful computer graphics engine (eg Unreal Engine 5) could achieve better and faster than the baselines utilized by this paper. In this regard, a state-of-the-art graphics pipeline could make the method in this paper less relevant, as they are slower and less controllable than traditional graphics software.\n- I believe the title \"Make Fashion Real\" is confusing. I understand what it tries to mean, but it is not clear and makes little sense.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for acknowledging the importance of the problem, the contribution of our dataset, the sound method and our presentation, the effective evaluation, and additional information in code and supplementary. The suggestions and questions also inspire us to improve our work. Below we address them in detail:\n\n> Some of the contributions of this work are very tied to a specific application. It is unclear how the proposed contributions, particularly the TAC could be leveraged for other applications. Are there any applications for which the proposed modules could be used, that are not specific to garment generation?\n\nIn general response, we further elaborate the scientific contributions of our work. In regard to other applications, our method can apply to more general rendered-to-real translation problems. We choose fashion images as they are the combination of highly articulated, complicated, structured, and meaningful content, which is also grounded in more direct commercial applications. In 3D games and animation, our method also have some potential if the problem of computational cost is mitigated in the future, as the reviewer mentioned and detailed in our discussion below. Besides, our method could also apply in other synthetic-to-real image generation applications, such as collecting data for model training in autonomous driving or robotics.\n\nThe proposed TAC module enables training-free control during the diffusion generation process. This feature allows for its application to a broader range of tasks, including text-to-image and image-to-image generation, such as personalized generation and image editing. As the number of model parameter becomes bigger and bigger, the training-free control can be utilized in an even wider variety of scenarios.\n> The analysis of the related work is somewhat lacking. Please see \"Questions\" for a few suggestions.\n\nWe thank the reviewer for suggesting these related works and will add discussion in our paper.\n\n> The proposed method is very computationally costly and makes it unsuitable for many applications, including interactive virtual try-on. \nA more powerful computer graphics engine (eg Unreal Engine 5) could achieve better and faster than the baselines utilized by this paper, and make the method in this paper less relevant.\n\nWe conduct some analysis on computational cost comparing to other SOTAs, in the rebuttal to Reviewer SbYx. Our method cannot handle real-time applications for now, but has potential for improvement with future integration with SD Turbo or SD Lightning. Note that comparing to other diffusion-base methods such as VCT, our method takes much less memory and time during testing as we do not need to perform additional optimization for each testing image.\n\nAs for rendering baselines, we build the 3D projects with Style3D Studio and use its integrated rendering tool based on rasterization. Using UE5 could potentially improve the rendering quality but will not diminish the effectiveness of our method. To verify this, we use more advanced rendering techniques via ray tracing (based on V-ray) to obtain rendered images, and our method consistently demonstrates its advantages in realism. Two visual examples are shown in Figure 2 of the PDF. We do not apply massive evaluation due to lack of time.\n\n> The paper is sometimes lacking in clarity, being overly convoluted in its writing. \nThe title \"Make Fashion Real\" is confusing.\n\nWe thank the reviewer for bringing up these issues and will refine our writing. We consider to change \"Make Fashion Real\" into \"FashionR2R\" for better reference to the main focus of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable time and insightful comments to help us improve our paper. In particular, we are encouraged by the positive feedbacks:
1. "The paper tackles an important problem". (Reviewer L3jv)
2. The proposed high-quality SynFashion dataset "fills a gap in relevant fields", "looks very interesting" and "is a salient contribution of this work", "facilitating further research in this area". (Reviewer m8pA, L3jv, SbYx)
3. "The proposed methods are sound and well-motivated", "novel", with "fairly standard components but combined in a sensible way". The TAC mechanism is "effective and crucial". (Reviewer L3jv, cE2m, SbYx)
4. "The framework impressively achieved transforming a clearly unrealistic image containing human faces into a plausible and realistic one". "Extensive experimental results demonstrate the superiority of the proposed method over existing techniques". "Curated qualitative results presented in the main paper are significantly than four baselines" and "quantitative metrics on two synthetic datasets are better". (Reviewer m8pA, SbYx, cE2m, L3jv)
5. "The paper is generally well-written, well-structured, and clear". "The proposed methods are well-motivated and well-presented." (Reviewer cE2m, L3jv)

Additionally, we are also grateful and very much inspired by the suggestions and questions. The shared concerns are about the academic/scientific contribution of this work (Reviewer m8pA, cE2m) and more robust quantitative evaluation (Reviewer SbYx, cE2m). Below we address these issues in detail:

> The academic/scientific contribution of this work.

1. This paper studies an important scientific problem. Modeling and producing lifelike clothed human images has attracted researchers' attention from different areas for decades. 

    ·  In computer graphics, researchers decompose and model the imaging process of a camera in real physical world. The related variables include 3D geometry, material (texture, reflectance, elasticity, etc.), illumination and camera parameters. All these variables are crucial and put into ray tracing algorithm to obtain the final image. To achieve photorealistic rendering, the variables and computation need to be as accurate as possible. However, this takes enormous time when the content is quite complicated, and may not be possible yet due to the limitation and somewhat plateaued ability of current techniques, especially for clothing. 

    ·  From the perspective of machine learning, researchers directly model the distributions of clothed human images. This is an implicit way to model the complicated variables and can leverage abundant knowledge through training on large datasets. With recent advancements in generative AI, the learning-based methods has achieved impressive quality in generated human images, comparing to rendering-based methods. However, the well-known limitations of generative methods lie in its generation diversity, stability, and most importantly, controllability and editability. 

    ·  This paper intends to establish on the controlled basis of rendering, and propose to leverage rich knowledge from abundant data and generative power from learning methods, to reach the goal of lifelike clothed human generation. We believe this is a fundamental and difficult scientific problem, and deserves researchers' attention.

2. The proposed method raises some fundamental questions in regard to current diffusion-based methods. The diffusion-based methods have demonstrates their power in various image generation and translation problems, such as style-transfer and super-resolution as discussed in our related work. However, it is unknow before, whether it is able to model the gaps between variables in rendering and physical world. This is particularly difficult due to the high expectation on the realism of the output images and the consistency in all levels of information with the rendering, given no paired data for training.

    ·  Our proposed DKI novelly adopts a positive finetuning for generation and a negative embedding for guidance, which achieves the realism enhancement to some extent. Our TAC further imposes texture and structure preservation without sacrificing too much realism. 

    ·  We believe this is not only an effective and novel discovery, but also proposes some fundamental questions on diffusion models: What are the inherent knowledge encoded by the UNet attention layers? Can we disentangle these structure to somehow explicitly model the gaps between variables in rendering and physical world? These questions might be non-trivial to answer, but our work would be an inspiration for future research in this area.

3. The proposed method is sound, the results are impressive and significant, and the dataset is a salient contribution, as acknowledged by the reviewers. We establish a comprehensive benchmark to help pave the way for future works to look deeper into remaining problems.

4. Our work is grounded in real-world applications, which is close to commercial influence and can potentially attract more resources to help develop this area. We believe this could also be beneficial to our NeurIPS community.

> More robust quantitative evaluation.

As suggested by Reviewer SbYx and cE2m, we have added user studies and conducted LPIPS as quantitative evaluation. The detailed results are provided in the individual response. 

We also include experiments on the textual related features as suggested by Reviewer m8pA, more advanced rendering method as suggested by Review L3jv and detailed visual ablation study on source DKI and target DKI as suggested by Reviewer cE2m, please refer to the attached PDF file.

As for other suggestions and questions, we answer them in the rebuttal to each reviewer individually. We sincerely look forward to further discussion and feedbacks, and thank you for your time in advance!

Meta-review: After the rebuttal, most reviewers agree that the paper tackles an important problem, the presented dataset will stimulate the future research along this direction. In the meantime, the algorithm proposed in this paper is effective, motivated, and extensive experiments are conducted to verify the proposed approach. The paper is well-written.

Agreeing with the reviewers, the AC accepts the paper, and encourages the authors to include the discussion, results presented during the rebuttal stage to the paper.
Decision: Accept (poster)