Image Translation as Diffusion Visual Programmers
Authors: Cheng Han
James Chenhao Liang
Qifan Wang
MAJID RABBANI
Sohail Dianat
Raghuveer Rao
Ying Nian Wu
Dongfang Liu
Venue: ICLR 2024 poster (2024)
Keywords: Image translation
Diffusion model
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Abstract: We introduce the novel Diffusion Visual Programmer (DVP), a neuro-symbolic image translation framework. Our proposed DVP seamlessly embeds a condition-flexible diffusion model within the GPT architecture, orchestrating a coherent sequence of visual programs ($i.e.$, computer vision models) for various pro-symbolic steps, which span RoI identification, style transfer, and position manipulation, facilitating transparent and controllable image translation processes. Extensive experiments demonstrate DVP’s remarkable performance, surpassing concurrent arts. This success can be attributed to several key features of DVP: First, DVP achieves condition-flexible translation via instance normalization, enabling the model to eliminate sensitivity caused by the manual guidance and optimally focus on textual descriptions for high-quality content generation. Second, the frame work enhances in-context reasoning by deciphering intricate high-dimensional concepts in feature spaces into more accessible low-dimensional symbols ($e.g.$, [Prompt], [RoI object]), allowing for localized, context-free editing while maintaining overall coherence. Last but not least, DVP improves systemic controllability and explainability by offering explicit symbolic representations at each programming stage, empowering users to intuitively interpret and modify results. Our research marks a substantial step towards harmonizing artificial image translation processes with cognitive intelligence, promising broader applications.

Published as a conference paper at ICLR 2024
IMAGE TRANSLATION AS DIFFUSION VISUAL PROGRAMMERS
Cheng Han & James C. Liang" & Qifan Wang & Majid Rabbani* & Sohail D & Raghuveer Rao & Ying Nian Wu & Dongfang Liu
Rochester Institute of Technology', University of Missouri Kansas City?, Meta AI
DEVCOM Army Research Laboratory', University of California, Los Angeles
ABSTRACT
We introduce the novel Diffusion Visual Programmer (DVP), neuro-symbolic image translation framework. Our proposed DVP seamlessly embeds a condition- flexible diffusion model within the GPT architecture, orchestrating coherent se- quence of visual programs (i.e., computer vision models) for various pro-symbolic steps, which span Rol identification, style transfer, and position manipulation, facilitating transparent and controllable image translation processes. Extensive experiments demonstrate DVP's remarkable performance, surpassing concurrent arts, This success can be attributed to several key features of DVP: First, DVP achieves condition-flexite model to eliminate sensitivity caused by the manual guidance and optimally focus on textual descriptions for high-quality content generation. Second, the frame- work enhances in-context reasoning by deciphering intricate high-dimensional concepts in feature spaces into more accessible low-dimensional symbols (e.g.. [Prompt], [Rol object]), allowing for localized, context-free free editing while main- taining overall coherence. Last but not least, DVP improves systemic control- lability and explainability by offering explicit symbolic representations each programming stage, empowering users to intuitively interpret and modify results. Our research marks a substantial step towards harmonizing artificial image trans- lation processes with cognitive intelligence, promising broader applications. Our demo page is released at here
translation via instance normalization, enabling the
INTRODUCTION
Concurrent state-of-the-art image translation methods predominantly follow the paradigm of connectionism (Garcez et al. (022). setting they aim
Imput [Easindtion] "change micdle duckv right kittien to pig"
these objects?
Oulput
went
"dwaep"
their goal as answering "what" to translate the image from the source domain to the designated target domain with fidelity. These methods can be broadly categorized into Generative Adversarial Network (GAN)- based and Diffusion-basel methods (see $2). Diffusion-based approaches usually demon- strate superior capabilities when compared to their GAN-based counterparts, especially in the domains such as image quality and coher- ence, training stability. and fine-grained con- trol (Dhariwal & Nichol| 2021). As such, they are perceived to harbor more auspicious poten- tial in image translation (Mokady al.) 2023).
[maage]
"right "pig"
Figure 1: Working pipeline showcase. DVP rep- resents a solution rooted in visual programming, demonstrating pronounced capabilities in-context reasoning and explainable control, in addition to its remarkable efficacy in style transfer.
Equal contribution Corresponding author

---

Published as a conference paper at ICLR 2024
Despite their huge success (Epstein et al. 2023: Dhariwal & Nichol 2021 these diffusion-based
methods exhibit a series of limitations: 1 Conditionn rigid learning. Existing arts following the
principles in classifier-free guidance (Ho & Salimans 2021;
202I:Mokady Mokady al. 2023) face significant
challenge in effectively achieving a harmonious equilibrium between unconditional and conditional predictions. Typically, these methods rely on manually crafted guidance scale parameters to pervise the process of each individual image translation. This inherent restriction curtails the algo- rithmic scalability, thereby impeding their potential for achieving holistic automation for real-world applications; @ Context-free incompetence. Concurrent methods primarily engage in the global ma- nipulation of diverse attributes (e.g., stylistic and content-related elements) within images, which prioritize the maintenance of contextual integrity over local alterations. Nevertheless, the absence of context-free reasoning hinders the precision required to realize specific Rol modifications, while simultaneously upholding broader sense coherence. Such contextual understanding capacities ne- cessitate a high degree of semantic acumen and a robust mastery of image structure, which is sorely lacking within contemporary image translation solutions (Hitzler et al. 2022 Oltramari et al. 2020 methods Mao et al. 2019). @ System opacity. Due to their black box nature, diffusion-
coined by connectionism (Wang et al., 2023) often exhibit a level of abstraction that distances them from the intrinsic physical characteristics of the problem they aim to model (Angelov & Soares, 2020). As result, users encounter limited control over the model's behaviors before reaching the final outputs, rendering them rudderless in establishing trustworthiness in decision decision-making or pur- suing systemic enhancement.
In this work, we revolutionize the focus of "what" into more flexible image translation paradigm of "where-then-what. " Under this paradigm, we first answer the question of "where" by finding the instructed target region accurately. After getting the target region (i.e., Region of Interest (Rol)), we answers the question of "what" " by translating it into the targeted domain with high-fidelity. Con- sidering the aforementioned discussions, we are approaching the task of image translation from fresh neuro-symbolic perspective, presenting a novel method named the Diffusion Visual Program- mer (DVP). More concretely, we architect an condition-flxxib harmoniously integrated with the foundation of GPT (Brown et al. (020). which orchestrates a con- catenation of off-the-shed computer vision models to yield the coveted outcomes. For example in Fig. D given an input instruction that specifies the translation from the "left dog" to a "sheep." DVP utilizes GPT as Al agent plan a sequence of programs with operations, subsequently invoked in the following procedure. It first addresses the fundamental question of "where" by identifying and segmenting the Rol of the "dog." " Once segmented, the background undergoes an inpainting to restore the regions obscured by the "dog," and our condition-flexible diffusion model translates स addressing the query of "what." DVP, leveraging spatial data, positions the "sheep" in the instructed spatial location, and further enables various context-free manipulations (see $3.2).
diffusion model (Song al.) 2021),
DVP is an intuitive and potent image translation framework, showing superior performance qual- itatively and quantitatively ($4.2) over -fate-of-the-art methods (Chen et al. 2023 Batzolis et al. 2021|Choo et al. [021] Rombaac et al. [2022] Saharia et al. [202]] Ruiz et al., 2023] Gal et al.]|202] Hertz et al. 2023 Mokady et al. 2023). enjoys several attractive qualities: Condition-flexibl translation. Our proposed condition-flexible diffusion model creatively utilizes instance normaliza- tion guidance (see $3.I) to mitigate the global noises in the unconditional embeddings by adaptive distribution shifting, and ensure that the model remains optimally conditioned based the textual descriptions without any parameter engineering. It offers a streamlined solution tackling the chal- lenge of hand-crafted guidance sensitivity on condition-rigid learning. This innovation addresses the "what" inquiry and enables a generalized learning paradigm for diffusion- solutions. @ Effective in-context reasoning. By decoupling the high-dimensional, intricate concepts in feature spaces into low-dimensional, simple symbols (e.g.. [Prompt), [Rol object]). we enable context-free manipulation of imagery contents via visual programming (see $3.2). It essentially includes a se- quence of operations (e.g.. segmentation, inpainting, translation) to establish in-context reasoning skills. Such a neuro-symbolic design fortifies our method with the capability to discern the con- cept of "where" with commendable precision. @ Enhanced controllability and explainability. Our modulating scheme uses explicit symbolic representations at each intermediate stage, permitting humans to intuitively interpret, comprehend, and modify. Leveraging by the step-by-step pipeline, we present not only a novel strong baseline but also a controllable and explainable framework to the community (see 4.3). Rather than necessitating the redesign of networks for additional functions or performance enhancement, we stand out for user-centric design, facilitating the seamless integration of future advanced modules.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Image-to-Image Translation. Image-to-lmage (121) translation aims at estimating a mapping of an image from source domain to target domain, while preserving the domain-invarian context from the imput image (Tumanyan et al., 2023] Isola et al. (017). For current data-driven methods, the 12I tasks dominated into two groups: GAN-based and Diffusion-bbsed methods. GAN-based methods, though they present high fidelity translation performance (Kim et al. [022] Park et al.] 22000a bj Zhu al. 2017a), pose significant challenges in training (Arjovsky et al, Gulrajani et al. 2017). and exhibit mode collapse in the output distribution (Metz et al,
2017
2017
Ravun & Vinyals 2019). In addition, many such models face constraints in producing 2023). Diffusion- seffussio-bbaedd methods, on the other hand, have recently shown competitive performance on generating fig-fiieliity images. Works on conditional diffusion models (Choi et al., 2021, Rombach et al., [022]] /Shharia et al., (202)) show promising performance, viewing conditional image generation through the integration of the reference image's encoded features during inversion (Mei et al.) (022). Though highly successful, they only offer coarse guidance on embedding spaces for generations (Chen et al., 2023), and exhibit ambiguity in intricate scenarios. In contrast, our approach, empowered by in-context reasoning, decouples complex scenes into low-dimensional concepts, and naturally handles such challenges.
diverse translation outcomes (Li et al.
Text-guided Diffusion Models. Several concurrent works contribute to text-guided diffusion models. DreamBooth (Ruiz et al. 2023) and Textual Inversion (Gal et al. (022) design pre-trained text-to-image diffusion models, while several images should be provided.
Prompt-toPrrompt (Hertz et al., 2023] Mokady et al., (023) manipulate local or global details solely by adjusting the text prompt. Through injecting internal cross- -oossatttention maps, they maintain the spatial configuration and geometry, thereby enabling the regeneration of an image while modifying it through prompt editing. However, (Hertz et al., 2023) does not apply an inversion technique, limiting to synthesized images (Mokady et al. 2023). They both introduce extra syper-parameter w (i.e., guidance scale parameter), which significantly influences the translated performance (see Fig. (5). In the light of this view, our approach re-considers the design of classifier-free guidance (Ho & Salimans, 2021) prediction, and further gets rid of such an oscillating parameter for robust prediction in a fully fully-auoomatic manner. We further include our intriguing findings in $4.3
Visual Programming. Visual programming serves as an intuitive way to articulate programmatic operations and data flows in addressing complex visual tasks (Gupta & Kembhavi 2023), coined by the paradigm of neuro-symbolic. Concurrent visual programming, empowered by Large Language Models (LLMs), shows superior performance in various vision tasks (e.g., visual relationship understanding (Donadello et al.] 2017) Zhou et al. 202T). visual question answering (Yi et al. 2018 Mao et al., | Amizadeh al. 2020), commonsense reasoning (Arabshahi et al.] 2021). image translation (Gupta & Kembhavi (203)). holding potential to change the landscape of image translation terms of transpurency and explainability, and in-context reasoning. We recognize number of insightful approaches, where the most relevant ones to our work are AnyDoor (Chen) cet al. 2023), Inst-Inpaint (Yildirim et al. 2023) and VISPROG (Gupta & Kembhavi 2023).
Nevertheless. (Chen et al. (023) select object to scene at manually selected location, classifies objects by an additional ID extractor, and extract detail maps for hicrarchical resolutions. (iildirim et al., 2023) needs additional training GQA-Inpaint dataset (Pfeiffer al., 2022).
Overall, these processes remain opaque or need additional training. which are not amenable to manual modifications or training-free paradigm. (Gupta & Kembhavi (023)} introduces visual programming to compositional visual tasks. However, it mainly contributes to the paradigm of building a modular neuro-symbolle system, while the strong editing abilities for compositional gen- eralization, has been overlooked. In addition, it simply replaces the original object by text-to -o-i-to-mmae generation, ignoring the preservation from any content from the original image. Our DVP, on the other hand, offers nuanced framework that delineates each intermediate step in a manner that is both comprehensible and modifiable for human users (see $4.3.. By translating complex feature spaces into lower-dimensional symbols, DVP enables fruitful context-free manipulations (sce $3.2)
3 APPROACH
In this section, we introduce Diffusion Visual Programmer (DVP), visual programming pipeline for image translation (see Fig. 2 Within our framework, image translation is decomposed into two distinct sub-objectives: 1 style transfer, translating Rols within images while upholding con- textual coherence; and @ context-free editing, endowing the capacity for unrestricted yet judicious modifications. In response to D, Condition-flexible diffusion model is introduced for autonomous,
3

---

Published as a conference paper at ICLR 2024
"change burger apple and then move to
Condition- flexible Diffusion Model
right/enlarge it." [lastructio]]
[Image]
[Rol Object]
Translated object)
Position Manipulator
Rol Segmentor
GPT-ériven Planner
Lapsinter
an apple
"an ice-cream and e-buegee
SSeenaiio
Preszlae
floating on ocean." [Prompt]
Plan
move apple right" or "enlarge the apple"
[Output]
Planner
Segmenter
Promptor
Inpointer
Menipulator
[Pap [Panlact Sonlection liage,
(Rel Dajed [[inage]
|Pappeed lnatrctoo),
[Scaaaoo]
[ girect), (Scaaano), "ooiton (Pe (SSte))
[Punil
Figure 2: Diffusion Visual Programmer (DVP) overview. Our proposed framework contains two core modules: is the condition-llexibl diffusion model (see (3.1). augmented by the integration of instance normalization (see Fig. 3. aimed to achieve more generalized approach to translation; stands for visual programming (see $3.2), fulfiiled by series of off off-the-shelf operations (e-g., Segment operation for precise Rol segmentation). The overall neuro-symbolic design enables in- context reasoning for context-free editing. We also enjoy enhanced controllability and explainability by intuitively explicit symbols (e.g., [Prompt], [Rol object], [Scenario], (Translated object)) at each intermediate stage, facilitating human interpretation, comprehension and modification.
non-human-intervene translation (.11). To achieve 2, we present In-context Visual Programming, which decomposes high-level concepts into human- uannaanndddrstadaalle symbols, enabling adaptable ma- nipulation ($ (..2). We claborate on techniques below.
3.1
CONDITION-FLEXIB DIFFUSION MODEL.
Preliminaries. Text-guided diffusion models transform stochastic noise vector 2, and a textual prompt embedding P into an output image aligned with the given conditioning prompt. For achieving step-by-step noise reduction, the model cg is calibrated estimate synthetic noise as:
min Ezz,ee~N(0..)....... tnin:.- (, Eo (Zz, P)]|?
(I)
where P is the conditioning prompt embedding and 2, is hidden layer perturbed by noise, which further introduced to the original sample data 2p based on the timestamp t. During inference, the model progressively eliminates the noise over T steps given noise vector 2r. To accurately reconstruct a given real image, the deterministic diffusion model sampling (Song et al. 2021 is:
(V2L1-- 0/-1
2t
Zt
-
,,0(2,,,
a4
for noise schedule. Motivated by (Hertz et al. 2023), we generate spatial attention maps corresponding to each textual token. Specifically, a cross-attention mechanism incorporated for controllability during translation, fa- cilitating interaction between the image and prompt during noise prediction as:
where 0a
(1-1- B:), and (0,1) 1) is
ep Zt, 1 P) = Softmax (((o(z=))] (v)()) (v)()),
(3)
Vd
where 1q. 1K. 1y are learned linear projections. 6(z)) represents spatial features of the noisy im- age, and (P) stands for the textual embedding.
Instance Normalization Guidance. gext-g generation frequently faces the challenge of mag- nifying the influence of the conditioning text during the generation process (Song et al. 2021). (Ho) Salimans| (021) thus put forth classifier-free guidance approach, making an initial prediction generated without any specific conditioning. This unconditioned output with the guidance scale pa- rameter w then linearly combined with predictions influenced by the conditioning text with scale (1 w). Formally, given Ø +b(-) as the feature representation from the null text, have:
tz,,,,
w 60 1, P) + (1 w) 6o =50 23)

---

Published as a conference paper at ICLR 2024
In practice, we observe that the scaling factor w is highly sensitive. Even minor fluctuations in its value can lead to substantial effects on the final images. This stringent requirement for fine-tuning on per-image basis makes its widespread adoption in real-world applications impractical. In the light of this view, we introduce the concept of adaptive distribution shift for the condition-llexibe translation. Specifically, we carefully consider the role of the two types of embeddings (i.e., the con- ditional prediction, E2 t, P) and the unconditional noise prediction eg t, 2) ). We argue that these embeddings warrant differentiated treatment, transcending rudimentary linear combination from Eq. 4 In the works of (Ulyanov et al. 2016; 2017), it is posited that the efficacy of instance normalization in style transfer can be attributed to its robustness against fluctuations in the content image. Classifier-free guidance (Ho & Salimans [ exhibits a similar form, which blends stable predictions from a conditional embedding with those concurrently trained unconditional embedding. We, therefore, utilize instance normalization and find that it effectively aligns with the in- tended translation direction by reducing the im- pact of unconditional embeddings (see Fig. 3 This ensures the diffusion model remains ex- clusively conditioned on the prompt, eliminat- ing any undesired variations. The introduction Figure 3: Instance Normalization Guidance.
conditional
DM
EB
Instance- normalization (see Eq. 5)
g
DM
uncordditonal
of instance normalization not only enhances the translation performance but also fortifies the model's ability to handle variations from the input distribution, regardless of potential differences in uncon- ditional distributions (see Fig. 5 Formally, the instance normalization guidance is formulated as:
(z( P, (co t, 2c)cooo( co(2e, L, 2)) (, P)), + ((coo=z,tt (5)
a(co(2,,t,))
where conv represents a 1 1x1 conv layer. Intuitively, the network ég is rcaled with o and shift with u,, where μ and a represent the mean of the unconditional embedding and the standard deviation of the conditional embedding, respectively. Given that both jı and a are known values from predictions, we elegantly abstain from operations that are influenced by human intervention for further uncondi- tional textual embedding tuning. Ablative studies in $4.3 further demonstrate our contributions.
3.2 IN-CONTEXT VISUAL. PROGRAMMING
Condition-flexible diffusion model in $3.1 provides generalized image translation solution, ad- dressing the "what" inquiry as the neural embodiment part in our proposed framework. We further present in-context visual programming, using a symbolic reasoning process to bridge the learning of visual concepts and text instructions, understanding the concept of "where" (see Fig, 6). By break- ing down the rich complexity of high-ddmeesiioal concepts into simpler, low-dimensional symbolic forms, we set the stage for a cascade of logical operations. This, in turn, enables the development of nuanced, in-context reasoning capabilities. The core components are articulated below.
Symbols. Within our framework, we produce intermediate steps such as [Prompt], [Rol object], [Scenario]. These intermediary outcomes not only facilitate transparency and explainability for hu- man interpretation, comprehension, and modifications, but can also be considered as symbols (Cor-) nelio et al. 2022). These symbols serve as -iw-dimensional data enable structured operations, bridg- ing the gap between raw data-driven methods and symbolic reasoning.
Operations. We carefully grounded five operations, {GP lan, PC, Segment, Inpaint, PM), in in-context visual programming (see Fig. 2). We first leverage the power of large language model -PTT4 (OpenAl 2023) as an AI agent to perceive, plan and generate program directives (Jen-) 2000 Anderson al. 2018). This component, named as the GPT-driven Planner module, manages corresponding programs with few examples of similar instructions, and invokes requisite operations via the GP lan operation in the reasoning process. All programs adhere to the first-order logic principles to identify attributes, allowing us to express more complex statements than what can be expressed with propositional logic alone (Mao et al. 2019). Specifically, these programs have a hierarchical structure of symbolic, functional operations (i.e., PG, Segment, Inpa int, PM), guid- ing the collections of modules (i.e., Prompter, Rol Segmenter, Inpainter and Position Manipulator) for each phase. Note that the operations can be executed in parallel, offering flexible combinations for systemic controllability via the program execution (see Fig. 7]. Specifically. Prompter utilizes any given input image by the operation PG. Our
nings
GPT-4 to generate detailed image descriptions input image therefore is not restricted solely to auman-annotated images as in previous arts (Mokady)
2023 et al.
Ho & Salimans! 2021 Chen al. 2023); rather, it enables random unlabeled images as
5

---

Published as a conference paper at ICLR 2024
Input Image
DVP (ours)
YQGAN-CLIP
Text2Live
SDEDIT
DillaselT
Prompt2Prompt
VISPROG
young rabet
"Chunye plane scrotch to plore
th the sendcarte
"Change the phato"
might fex erctio woll"
"Cherge the randon cranges
"Change focal to razl
Figure 4: Qualitative results with the state-of-the-art baselines. DVP exhibits rich capability in style transfer, achieving realistic quality while retaining high fidelity. Owing to the context- free manipulation (see $3.2). the DVP framework is capable of flawlessly preserving the background scenes while specifically targeting the translation of the Rol. Note that while VISPROG also enables context-free editing, it exhibits considerable limitations in rational manipulation (see Fig. 6
inputs, thereby broadening the potential application scenarios and augmenting data efficiency. Rol Segmenter creates flexible Rol segmentation via pre-trained Mask2former (Cheng et al. 2022). in alignment with the operation of Segment. Inpuinter operates based on the logic of Rol, in- dicating the completion of foreground/background via stable diffusion v1.5 (Rombach et al. 2022
Lugmayr al. (0022). assigned the Inpaint operation. We further classify rational concepts (e.g., position, scale) by Position Manipulator (i.e., assigned as the PM operation), translating the programmed human language instructions into Nomain-speciifc language (DSL) designed for po- sitioning. The DSL covers a set of fundamental logic such as Enlar arla,,e, Shrink, Left, Right, Up. and Down etc. These intuitive logic share the same input format and output interface, thus achieving programming flexibility (see Fig. 7
Program Execution. The programs are managed by the Compiler (Gupta & Kembhavi 2023). which creates mapping association between variables and values. It further steps through the pro- gram by incorporating the correct operations line-by-line. At every stage execution, the program will activate the specified operation, and the intermediate outputs (e.g.. [prompts], [Rol object] in human-ine iamaaninteerrretbble symbols, thereby enhancing the explainability of the system. Since these out- puts enable direct visual evaluations, offering an option either repeat the current step or proceed to the next one, which in turn improves the system's 's controllability, demonstrated in $4.3
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Benchmarks. For quantitative and qualitative results, we conduct a new benchmark (see SE) con- sisting of 100 diverse text-image pairs. Specifically, we manually pick images from web, generated

---

Published as a conference paper at ICLR 2024
images, ImageNet-R (Hendrycks et al. 2021). ImageNet Ruussakvssyy et al. 2015 MS COCO Lin et al., 2014). and other previous work (Ruiz et al.) 2023 Tumanyan et al. 2023). To ensure a fair companson and analysis, we construct a list of text templates including 20 different classes for trans- lating, each containing five images of high-resolution and quality. For each originating class (e.g.. building), we stipulate corresponding target categories and stylistic attributes thereby facilitating the automated sampling of various permutations and combinations for translating and evaluation. Evaluation Metrics. We follow (Ruiz al. [003]] Chen al. 2023), and calculate the CLIP-
Score (Hessel et al. (02T) and DINO-Score (Caron et 20211) These metrics enable the similarity between the generated image and the target prompt. We further conduct user studies involving 50 participants to evaluate 100 sets of results from multiple viewpoints, such as fidelity, quality, and diversity. Each set comprises source image along with its corresponding translated image. To facilitate the evaluation, we establish comprehensive guidelines and scoring templates, where scores range from (worst) to 5 (best). Please refer to S for more details.
4.2 COMPARISONS WITH CURRENT METHODS
Qualitative Results. For fair comparison, we include six state-of-the-art baselines (i.e.. SDEdit (Meng al., (2022), Prompt-to-Prompt (Hertz al. 2023), DiffuselT (Kwon & Ye 2023).
2022) and VISPROG (Gupta & Kembhavi 2023) on diverse prompt-guided image translation tasks. We present the visualization For style transfer, incorporating instance normalization imbues our translated images strike a balance between high preservation to the guidance layout, and high fidelity to the target prompt. For example, DVP vividly transfers the "young boy". "church", etc. into "robot", "sandcastle", etc. in Fig. 4 Other methods contrastingly yields corrupted translation on these Rol or introduce excessive variations. DVP also exhibits strong in-context reasoning by acknowledging and transferring objects as instructed by the user. For example, DVP selectively modify the "right fox" to the "aretic wolf," while leaving other foxes and the background unaffected (see Fig. Jiith row). Though VISPROG enables local editing, it falls short in cultivating the context reasoning skills, and result in translating all "foxes" into "aretic wolves" (see $2). The overall results demonstrate that SDEdit and VQGAN-CLIP compromises between preserving structural integrity and allowing for significant visual changes. DiffuselT and Prompt2Prompt maintain the shape of the guidance image but makes minimal alterations to its appearance. Text2Live exhibits associations between text and certain visual elements (e.g., the shape of objects); however, translating the objects to entirely new ones may not lead to visually pleasing results. In contrast, DVP effectively handles all examples across various scenarios and instructions.
VQGAN-CLIP (Crowson et al., 2022). Tex12LIVE (Bar- Tal et al.
results with DVP in Fig.
Quantitative Comparisons. Dis- cussed in | we further report the user study, CLIP-Score, and DINO-Score in Table [ The re- sults are consistent with our visual evidence: DVP distinctively out- performs other competitors in both fidelity and quality, most notably in the realm of fidelity. While existing methods are designed to consider merely on the seman- tic consistency, DVP goes one step further by introducing visual programming understanding with cognitive reasoning that preserving instance identity and background scenarios. This inherently allows for superior performance in terms of fidelity, thereby making our model more versatile and applicable in various scenarios.
Inport Image
Tedance Narn
$2-25
W45
W=10
Por the
Figure 5: Ablative visualization results of instance normal- ization compared with various guidance scales w.
4.3 SYSTEMIC DIAGNOSIS
This section ablates core design of DVP instance normalization guidance in translation ($31) and in-context reasoning in visual programming ($3.2). Our systemic merits in explainable control- lability as well label efficiency, coined by the neuro-symbolic paradigm, also discussed.
7

---

Published as a conference paper at ICLR 2024
Table 1: User study, CLIP-Score, and DINO-Score on the comparison between our proposed
framework and ofate-of-the-art baselines. The metrics are detailed in $4.1
User Study
Quantitative Results
Method
Quality
Fidelity
Diversity
-LoP-Score
DINO-Score
VOGAN-CLIP Crowson et al. 2022)
3.25
3.16
3.29
0.749
0.666
Text2Live (Bar- lal et al. 2022)
3.55
3.45
3.73
0.785
0.659
SDEDIT Meng et an, 2022
3.37
3.46
3.32
0.754
0.642
Prompt2Prompl (Hertz ef al. 2023)
3.82
3.92
3.48
0.825
0.657
DiffuselT (Kwon 2023)
3.88
3.87
3.57
0,804
0.648
VISPROG (Gupta & Kembhavi 2023)
3.86
4.04
3.44
0.813
0.651
DVP (ours)
3.95
4.28
3.56
0.839
0.697
"Olenge You right
"chorae the left
laput Image
DVF (U USS)
FropptProap
lapal Image
DVF (0urs)
VISPROG
"Sweo the staffee stuffed lae"
and the eiephor"
and siepoal
Athertor RED ETSPETT
menley and the
elenect meal
"Chinge the shuff
Iapat Image
DVP (0 urs)
PrompC1Proapp
laput Image
DVP (Durs)
VISPROG
Figure Visualization results of in-context reasoning against attention-based Prompt2Prompt
Mokady et al. 2023) and programming-bisn based VISPROG (Gupta & Kembhavi 2023)
Instance Normalization. Our condition-flexibee diffusion model diverges from conventional ap- proaches (Mokady et al. [2023] Ho & Salimans] (021) by employing instance normalization guid- enhance the robustness of translations and capacity manage variations in the input dis- tribution (see $3.1). To verify diffusion model methods with instance normalization and guidance scale (Mokady al. 2023), we conduct extensive experiments in Fig. and Table 2 a) qualitatively and quantitatively. For fairness, these comparisons are made without incorporating in-context visual programming into our approach. We opt for the guidance scale parameter range centered around 7.5 for linear combination, which is chosen based on the default parameter provided in (Mokady) al. 2023, Ho Salimans 2021). The parameter is varied with a step size 2.5 to investigate its influence the translated images. While the guidance scale baseline exhibits substantial variations on translated images (see Fig. right 4 columns) when w E (2.5, 5, 7.5, 10), instance normalization enables high fidelity with natural, stable translation without any manually tuned parameter.
In-context Reasoning. DVP em- ploys set of visual programming operations for image translation, powerful
"Change the woman to an astronaut"
"Change left horse zebra"
thereby facilitating in-context reasoning capability during image manipulation. To better demonstrate our claim, we present and compare our translated results with strong cross- attention map baseline method Prompt2Prompt (Mokady al. 2023) in Fig. 6 Specificallly, in the first of Fig. 6 aim
Instuction
Indortion
[Plan] #G ([Imáge), [Instriction]
Plan] an ((imdge), [Instniction]]
Plan-1
Plan-2
Plan-1
Plan-2 Trinsdaed
Dal mad gaesst
Bel Chacc madoonin
Edag
condi
Tranaladd Oijlde
Cleck
Ditee, made
Toualaed Uijece
Rellonee rack
to translate only the right pigeon an cagle. The cross-attention map on Promp(2Prompt indicates that recognizes both pigeons, albeit with notable failure to discern the positional information accurately. Consequently, the model erroneously translates both pigeons into eagles. In contrast, our approach exhibits a strong in-context understanding of the
OMkssiofTiaaled Oegt.
Oppdlee Coanina
Oggate wwweeataaaaaee Traauaa Clack
Ogg Tesddailttaniee Coaee
the
Ouliptt
Figure 7: Explainable controllability during program execu- tion, which easy for error assessment and correction.
8

---

Published as conference paper at ICLR 2024
Table Ablative results on instance normalization guidance and label efficiency.
Metries
w 2.5
w
7.5
10
Inst. norm
Methods
CLIP CLIP-Score DINO-Scome
CLIP-Score
0.833
0.795
0.742
0.686
0.839
Human-annotated
0.817
0.688
DINO-Scorr Score
0.664
0.681
0.712
0.678
0.697
Prompter
0.839
0.697
(a) Instance normalization
(b) Label efficiency
scene and accurately translates the designated pigeon as instructed. We design more sophisticated
instructions in the second row example in Fig. For Prompt2Prompt, the ambiguous cross-attention on elephant results in wrongful artifacts across various animals in translation. It also demonstrates limitations in performing Rol relation editing (i.e., Swap, Enlarge) We further compare DVP with VISPROG (Gupta & Kembhavi, 2023). a visual programming approach that enables local image editing (see $2). As seen, although both DVP and VISPROG allow for instructive object replacement (Fig. 6 right section), VISPROG is inferior in translation fidelity (e.g., translated duck too yellow and lion has different pose). VISPROG also fails to support relation manipulations while DVP exhibits commendable result in all examples following strictly to human instructions. Explainable Controllability. We next study the explainable controllability during program execution (see (3.2). In design, enable multiple operations worked in parallel, there are different program plans available for a diverse order of operation sequences. As shown in Fig. 7 (left section), "change the left woman Throughout the execution process, the program is run line-by-line triggering the specified operation and yielding human-interpreetalle intermediate outputs each step. thereby facilitating systemic explainability for error correction. As shown in Fig. (right section), Plan-1 mistakenly segments only the head of the "left horse," leading an incomplete translation to "zebra." " Owing to the provision of explainable output at each step, we are able to identify specific issues and subsequently employ alternative Plan-2 as a more suitable strategy for translating the "left horse."
astronaut" can be achieved by using both Plan-1 and Plan-2.
Label Efficiency. Prompter generates detailed image de- scriptions for arbitrary input images, thereby relaxing label dependency without being tightly bound by human anno- tations (see $3.2. To verify its efficacy, we introduce user study (i.e., Table [))) and further provide visual evidences (i.e., Fig. 8), contrasting human- generated annotations with those produced by Prompter. Given that the focus of this ab- lative study to investigate the impact of labels on translated images, visual programming is excluded from the experimental design. In Fig. Otherwise, the reconstructed images would closely resemble the input images, irrespective of the prompts, in scenarios where the prompts entirely erroneous. As seen, the GPT-4 generated annotations result in superior performance in both CLIP-Score and DINO-Score (i.e., Table ((b)), indicating that the generated annotations benefit the quality of the translated images. The visual evidences in Fig. support our claim: by having detailed descriptions of images, the reconstruction phase yields finer results, which subsequently manifest in the quality of the final translated images.
Roconstracion
Tranditioon
"change the corgi
Ingat image
Astamatio bbel cany thinng in arsage beckgrousd"
Binker Look
"Change ofter ses Honn
Ingutinagg
Soo the othe"
by sta"
Figure Human-annotated and Prompter-generated descrip- tions. Descriptions generated via the Prompter yield finer recon- structions, as well as subsequent translated outputs, suggesting a relaxation of label dependency.
bypassed 60% of the optimization steps.
CONCLUSION
In this work, we introduce DVP, a neuro-symbolic framework for image translation. Compared to concurrent image translation approaches, DVP has merits in: i) generalized translation without con- sidering hand-crafted guidance scales on condition-rigid learning: ii) simple yet powerful in-context reasoning via visual programming; iii) intuitive controllability and explainability by step-by-step program execution and parallel operations. As whole, we conclude that the outcomes presented in our paper contribute foundational insights into both image translation and neuro-symbolic domains.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This research was supported by the National Science Foundation under Grant No. 2242243 and the DEVCOM Army Research Laboratory under Contract W911QX-21-D-0001.
REFERENCES
Elaine Allen and Christopher A Seaman. Likert scales and data analyses. Quality Progress, 40(7): 64 64-63, 2007.
Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang, and Kazuhito Koishida. Neuro- symbolic visual reasoning: Disentangling. In ICML, 2020.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting grouullygrounded navigation instructions in real environments. In CVPR, 2018.
Plamen Angelov and Eduardo Soares. Towards explainable deep neural networks (xdnn). Neural Networks, 130: 185-14,,, 2020.
Forough Arabshahi, Jennifer Lee, Mikayla Gawarecki, Kathryn Mazaitis, Amos Azaria, and Tom Mitchell. Conversational neuro-symbolic commonsense reasoning. AAAI, 2021.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, 2017.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. In ECCV, 2022.
Georgios Batzolis, Jan Stanczuk, Carola-Bibian Schönlieb, and Christian Etmann. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111.13606 2021.
Dane Bertram. Likert scales. Retrieved November, 1000):1-10,2000 2007.
Harry N Boone Jr and Deborah A Boone. Analyzing likert data. The Journal of Extension, 50(2): 48, 2012.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero- shot object-level image customization. arXiv preprint arXi::2307.0048// 2023,
Bowen Cheng, Ishan Misra, Alexander G Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR. 2022.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. ICCV, 2021.
Cristina Cornelio, Jan Stuehmer, Shell Xu Hu, and Timothy Hospedales. Learning where and when to reason in neuro-symbolic inference. In ICLR, 2022.
Katherine Crowson, Stella Biderman, Daniel Koriis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. In ECCV, 2022.
10

---

Published as a conference paper at ICLR 2024
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. NeurIPS, 2021.
Ivan Donadello, Luciano Serafini, and Artur D' D'Avila Garcez. Logic tensor networks for semantic image interpretation. IJCAI, 2017.
Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. arXiv preprint arXiv:2306.00986 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or, An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.014 01618, 2022.
Artur d' Avila Garcez, Sebastian Bader, Howard Bowman, Luis C Lamb, Leo de Penning, BV Illu minoo, Hoifung Poon, and COPPE Gerson Zaverucha. Neural-symbolic learning and reasoning: A survey and interpretation. Suuro-Symoolic Artificial Intelligence: The State of the Art, 2022.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. NeurIPS, 2017.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In CVPR, 2023.
Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E' 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.13770, 2023.
Cheng Han, Qifan Wang, Yiming Cui, Wenguan Wang, Lifu Huang, Siyuan Qi, and Dongfang Liu. Facing the elephant the room: Visual prompt tuning or full finetuning? arXiv preprint
1rXiv:2401.12902. 2024.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of -fitt--diitribution generalization. In ICCV, 2021.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. ICLR, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
Pascal Hitzler, Aaron Eberhart, Monireh Ebrahimi, Md Kamruzzaman Sarker, and Lu Zhou. Neuro- symbolic approaches in artificial intelligence. National Science Review, 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurlPS Workshop, 2021.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Nicholas R Jennings. On agent-based software engineering. Artificial intelligence, 2000.
Kunhee Kim. Sanghun Park. Eunyeong Jeon, Tachun Kim, and Daijin Kim. A style-aware discrim- inator for controllable image translation. In CVPR, 2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02644 2023.
Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. ICLR, 2023.
Bo Li, Kaitao Xue, Bin Liu, and Yu-Kun Lai. Bbdm: mmage-to-image translation with brownian bridge diffusion models. In CVPR, 2023.
11

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-
training for unified vision-language understanding and generation. In ICML, 2022.
Rensis Likert. A technique for the measurement of attitudes. Archives of Psychology, 1932.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR, 2021.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. ICLR, 2019.
Kangfu Mei, Nithin Gopalakrishnan Nair, and Vishal M Patel. Bi-noising diffusion: Towards con- ditional diffusion models with generative restoration priors. arXiv preprint arXiv:2212.07352, 2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. ICLR, 2022.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. ICLR, 2017.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023.
Geoff Norman. Likert scales, levels of measurement and the "laws" of statisties. Advances in Health Sciences Education, 15:625-632, 2010.
Alessandro Oltramari, Jonathan Francis, Cory Henson, Kaixin Ma, and Ruwan Wickramarachchi. Neuro-symbolic architectures for context understanding, arXiv preprint arX1v:2003.04777, 2020.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
URL https://api.
semanticscho.oo org/CorpuslD: 25:75332815
Taesung Park, Alexei A Efros, Richard Zhang, and Jun Yan Zhu. Contrastive learning for unpaired image-to-image translation. In ECCV, 2020a.
Taesung Park, Jun-Yas Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. NeurIPS, 2020b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin O. Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xGQA: Cross-Lingual Visual Question Answering. In ACL, 2022.
Zheyun Qin, Xiankai Lu, Xiushan Nie, Dongfang Liu, Yilong Yin, and Wenguan Wang. Coarse- to-fine video instance segmentation with factorized conditional appearance flows. JEEF/CAA Journal of Automatica Sinica, 2023.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. NeurIPS, 2019.
12

---

Published as a conference paper at ICLR 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning tex--to-image diffusion models for subject-driven generation. In CVPR, 2023.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. IJCV, 2015.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: image-to-image diffusion models. ACM SIGGRAPH, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. CVPR, 2023.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 2016.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In CVPR, 2017.
Wenguan Wang, James Liang, and Dongfang Liu. Learning equivariant segmentation with instance- unique querying. NeurIPS, 2022.
Wenguan Wang. Cheng Han, Tianfci Zhou, and Dongfang Liu. Visual recognition with deep nearest centroids. In ICLR, 2023.
Liqi Yan, Cheng Han, Zenglin Xu, Dongfang Liu, and Qifan Wang. Prompt learns prompt: exploring knowledge-aweed aware generative prompt collaboration for video captioning, In IJCAI, 2023.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. NeurIPS, 2018.
Ahmet Burak Yildirim, Vedat Baday, Erkut Erdem, Aykut Erdem, and Aysegul Dundar. Inst-inpaint: Instructing remove objects with diffusion models, 2023.
Ziheng Zhang, Anpei Chen, Ling Xie, Jingyi Yu, and Shenghua Gao. Learning semantics-aware distance map with semantics layering network for amodal instance segmentation. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 2124-2132, 2019.
Tianfei Zhou, Siyuan Qi, Wenguan Wang, Jianbing Shen, and Song-Chun Zhu. Cascaded parsing of human-object interaction recognition. IEEE TPAMI, 2021.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017a.
Yan Zhu, Yuandong Tian, Dimitris Metaxas, and Piotr Dollár. Semantic amodal segmentation. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit, 2017b.
13

---

Published as a conference paper at ICLR 2024
SUMMARY OF THE APPENDIX
This appendix contains additional details for the ICLR 2024 submission, titled "Image Translation Diffusion Visual Programmers." The appendix is organized follows:
Implementation details and the pseudo-code.
More quantitative results for instance normalization.
More quantitative results for in-context reasoning.
Video translation.
General-conter dataset.
Runtime analysis.
More qualitative results.
Module Replacement Analysis.
More comparison with Insiruct-Pix2Pix.
Ambiguous instruction analysis.
K
Complex instruction analysis.
Error analysis.
Introduction of user study metries and more user study results.
Discussion of limitations, societal impact, and directions of our future work.
A IMPLEMENTATION DETAILS AND PSEUDO-CODE OF DVP
Hyper-parameters. We follow the common practices (Ruiz et al. 2023 Hertz al., 2023 K Kwon & Ye 2023) and use stable diffusion v1.5 (Rombach et al. 2022) the base generator. For null- text optimization (Mokady et al. 2023), we process the image resolution to 512 512. choose AdamW (Loshchilov & Hutter 2019) optimizer with initial learning rate of 1e-5, betas (0.9, 0.999). eps = 1c-8 and weight decays 0.01 as default.
felditeon-flexible Diffusion Model. For all experiments, utilize diffusion model (Song al. 2021) deterministic sampling with 50 steps. The deterministic diffusion model inversion is per- formed with 1000 forward steps and 1000 backward steps. Our translation results are performed with 50 sampling steps, thus we extract features only at these steps. Within each step, null-text opti- mization (Mokady et 2023) is employed for inference time pivotal fine-tuning without additional training schedule. Specifically for each input image, our approach optimizes the x conv layer and the unconditional embedding concurrently. Furthermore, we follow the common practise (Mokady) et al. 2023 Hertz et al., 2023) and re-weight the positive and negative prompting via the hyper- parameter, which is used to magnifyldepreciate the attention during the conditional prediction and unconditional prediction process.
In-context Visual Programming. We choose the GPT-4 (OpenAI 2023) as our Planner discussed in $3.2 utilizing the official OpenAI Python API. The maximum length for generated programs is set to 256, and the temperature is set to O for the most deterministic generation. Furthermore, we utilize Mask2Former (Cheng et al.) 2022)
segmenter, Repaint (Lugmayr et al. 2022) painter, and BLIP (Li al. 2022) as a prompter.
in-
Competitors. We employ the official implementations provided for each competitor for fair com-
parisons: VISPROG (Gupta & Kembhavil 2023), Prompt-to-Prompl (Hertz et al.) (023), Diffu- seIT (Kwon& & Ye 2023). and Text2LIVE Bar-Tal et al. (2022). To run SDEdit (Meng et 2022) on stable diffusion, we utilize the code provided in the Stable Diffusion official repo The publicly accessible repository for VQGAN-CLIP (Crowson et al. (2022) also applied in our experiments. Algorithm and Pseudo-code. The algorithm of condition-flexible diffusion model (see $3.1) is demonstrated in Algorithm [ The pseudo-code of instance normalization guidance in condition- flexible diffusion model is given in Pseudo-code [ Modules for in-context visual programming are provided in Pseudo-code 2 Our work implemented in Pytorch (Paszke et al. 2019). Experi- ments are conducted Reproducibility. Our demo page is released at here. To further guarantee reproducibility, our full implementation and code publicly released.
NVIDIA TESLA A100-80GB SXM GPU.
14

---

Published as a conference paper at ICLR 2024
Algorithm 1: fonditioon-lexibbee diffusion model inversion
Input: source prompt embedding translation textual embedding P and input image Z.
Output: Noise vector 277.
Using instance normalization;
Compute the intermediate results ;.:....;0.... using diffusion model inversion over Z; Using instance normalization;
Initialize 2T f- 27,, ØT + (i("));
for f T, - 1,..., do
for j = 0, ......N
do
Z 12 R1 P)l12 2, 01 1/V V 1
end
Set 1 t- 2-, Øt. P), 10-1 + 23
Update 2i-1 t Eg [,, Ø3)
end Return 27
Pseudo-code 1: Pseudo-code instance normalization used in cundition-flexible diffusion model in PyTorch-like style.
noise: conditional noise prediction embeddings
uncond_noise: unconditional neise prediction embeddings
def calc_mean_std (input, eps-le-8):
4-- calculation of the mean and standard deviation of the input embedding
--1
size input.size
N, size
input_ var = irput. viput.view(h (N, C, -1) vvar sps
input_std feat_var, sqrt () view (N, 1,
input_mean input.view return iaput_meant iaput_std
view (N, C, -1) .mean (dim=2) (N, c, 1, 1)
instance norma izaaiion (cond_noised uncond_noise): :
#= Insnsance normall i zation of shifting the di distribution to conditional
noise prediction ==$
size coad_noise. size
incond_mean, uncond_std cale mean std (urcond_noise)
cord_maan, cond_std calc_rean_std (cond_roiss
normal ized_noise- cancon_nncise cond_mean. exeaadd size)) )(ee))/ccom) /cond_std.expant size)
return conv inwommliieednoiiiee.
second_std.expee (size)
uncond uncond_mean.eepand
(size)
B MORE QUALITATIVE RESULTS FOR INSTANCE NORMALIZATION
In this section, we provided more qualitative results for instance normalization guidance (see $3.1) compared with different guidance scales w in Fig 9 For fair comparison to other approaches, we do not utilize visual programming here (see $4.3) In addition, we report the overall qualitative results proposed inddition-flexible one within the framework of DVP. As shown in Fig. [T our results consistently achieve robust performance with high fidelity when comparing with the fixed scale approach.
by integrating either traditional diffusion model
15

---

Published as a conference paper at ICLR 2024
Pseudo-code 2: Pseudo-code of in-context visual programming in PyTorch-like style.
class VisualPxogrammtnngg def _if__int____ (self);
load pre-trained models and transfer them the GFU
def (self, f, inputs: Tist):
return an string that visually represents the input and output steps,
def (sele plan: str):
analyze the step, extracting a list of input values and variables along with the name the output variable
def crop (self, image, plan: str):
return the image cropping at the given coordinates and stores the coordinates as attributes
def find f, image, plan: str):
return 1ist of objects are discovered
match the object_name if any such objects
def move (self, lamge, objects: Dict, plan: stri:
zeturn an image with objects that have been moved to new positions
def execute (self, plan: str, state):
inputs, input_var_names, cutput_vvr_name self.parse (plan)
retrieve the values of input variables from the state
for var_name in iaput_ver_nett _ar__mess_m_m
inpats.
execute the code and calculates the output atilizing the loaded model output some_computatiog (topptaaiion(inputs)) (inputs)
update the state stateloutput
cutput
visual representation summarizing the computation of the step. step_htmi self.atmi (inputa, output)
return output, step_html
C MORE QUALITATIVE RESULTS FOR IN-CONTEXT REASONING
The DVP framework utilizes a series of visual programming operations enable image trans- lation, thereby enhancing its capacity for context-aware reasoning for arbitrary content manipu- lation. Fig. 10 provides more results and showcases our advantages over strong baselines, i.e., Prompt2Prompt (Mokady al. 2023) and VISPROG (Gupta & Kembhavi 2023).
D VIDEO TRANSLATION
We further evaluate the efficacy of DVP on video translation in Fig. 12 Empirically, we are profi- cient translating video content without the necessity for explicit temporal modeling. However, we do acknowledge certain issues during translation: a) the occurrence of artifacts, which depict non- existent clements within the original image (see D); and b) instances of missing objects (see D), often resulting from the failure accurately identify the object, possibly due to the object possess- ing only a limited number informative pixels or assuming an unconventional pose. More failure cases discussed in L
E GENERAL-CONTEET DATASET
We present general-context dataset consisting of 500 image-text pairs across 20 different classes representing common objects their typical contexts. Each class contains 25 unique samples. The objects span range of common everyday items (see Fig. B top) such as foods, furniture, ap- pliances, clothing, animals, plants, vehicles, etc. The images consist of high-quality photographs capturing the objects in natural environments and generated images with advanced digital rendering
16

---

Published as a conference paper at ICLR 2024
Input Image
Instance Norm
W=2.5
W=5
W
W=10
"Change the image to Mone? style"
"Cartsonize the photo onime style"
"Change the image to oil painting"
"honge Image
Figure 9: More qualitative results for instance-normalization
techniques. The textual descriptions detail the location and spatial connections between primary ob- jects and other context-rich elements in the scene produced by the GPT-driven prompter (see $3.2). Each final image-text pair (see Fig. 13 bottom) incorporates a source image with its descriptions, comprehensive instructions, and translated image generated by DVP, which also includes cues about the position of objects in relation to other elements in the setting (e.g., "Two foxes are stand- ing the right of wolf that perched on rock in front of a forest"). The fine-grained details in both images and instructions aim to establish a comprehensive and demanding benchmark for future studies capable of precisely anchoring position relations within intricate real-world scenarios.
F RUNTIME ANALYSIS
We further do runtime analysis, shown in Table 3 As seen, though SDEdit achieves a superior performance in editing speed by exclud- ing diffusson inversion process, coming with price that is less effective in retaining the intricate details from the original image. In contrast, proposed method achieves satisfying image transla- tion results with limited computational overhead (e.g., ~ 30%) when compared other famous techniques, i.e., Tex12Live (Bar-Tal al. 2022). VQGAN-CLA (Crowson et al. 2022). and VISPROG (Gupta & Kembhavi 2023). Specifically, Text2live (Bar- Tal al. 2022)
Table 3: Runtime analysis.
We measure the total editing time for different methods.
Method SDEdit
Editing
10s
VQGAN-CLIP
~ 1.5m
Text2live
10m
VISPROG
1.5m
generates an edit layer of color and opacity, which is then composited
Ours
2 m
over the original input image, rather than directly producing an edited output: VQGAN-CLIP (Crow-
son et al.) 2022) employs multimodal encoder to assess the similarity between a text-image pair and iteratively updates the candidate generation until it closely resembles the target text; VIS- PROG (Gupta & Kembhavi 2023) introduces visual programming to compositional visual tasks relying solely on a standard stable diffusion model (Rombach et al. 2022). Compared to these methods, DVP facilitates various editing operations (sce $3.2) during the translation process, adding to its practicality in applications where versatility in editing is crucial (see $4.2). We thus argue that the overhead of proposed DVP is acceptable, considering its satisfying performance.
17

---

Published as a conference paper at ICLR 2024
ange the left
Aftustin shp
Thagg the right
Laput HiDge
(OUUS))
Prompi2Proppt
Teput Image
DVF (ours)
Aftartie ruldo paroce
partiin a robee"
percon Tu rileer
Tapel Imsge
DVF (ours)
Prompt22rommp
Toput Image
IDU (ourd)
V VIsPROG
Aftestia
"Dagge ell aragg
Ispat imago
Promptarmmpl
reput Image
DV F (ourg)
V VISPIOON
Alteatinn lrgeet
bear puidil"
baur
Inpat tmagg
DVV P (nun))
Prompt2/rompi
Laput Image
DV2 (ours)
VISPROG
Figure 10: More qualitative results for in-context reasoning.
Input Image
Instance Norm
W=2.5
W
W=10
"Change: the apples in the basket to peaches"
Figure 11: Qualitative results by integrating either traditional diffusion model our proposed condition-flexibe one (see $.11.
G MORE QUALITATIVE RESULTS
We present more qualitative results in Fig. 14 As seen, DVP is capable of achieving appealing performance in various challenging scenarios. For example, the first row of Fig. [4] DVP trans- fers the husky into sheep with high fidelity, and preserves all details from the intricate outdoor background.
H MODULE REPLACEMENT ANALYSIS
We also introduce a segmentation model replacement module, substituting the model (Cheng et al. 2022) with Segment Anything (Kirillov al. 2023 As demon- strated in Fig. [5] Segment Anything exhibits superior segmentation capacity (correctly iden- tifying zebra with partial occlusion), resulting improved translation results. Our modu- lating design enhances overall performance by replacing the segmentation component, particularly in scenarios that demand accurate per-pixel understanding. This characteristic ensures the flexibility of our approach, enabling effective management of diverse range of scenarios Wang et al. 2022 Qin al. 2023
Mask
Editing
Input Image
the middie zebro
Figure 15: Module replacement analysis. We re- place Mask2former (Cheng al. 2022) to Seg- ment Anything (Kirillov et al. 2023 as our Rol Segmenter in DVP.
Liu et al. 2021; Han et al. 2023 an et al. 2023
Han et al. 2024
18

---

Published as a conference paper at ICLR 2024
1 COMPARISON WITH INSTRUCT-PiX2PIX
In Fig. 16 present detailed qualitative comparison between our method and Instruct- Pix2Pix (Brooks et al. 2023). As seen, our approach yields better qualitative results. For example, when changing the kid branches, DVP presents a more faithful translated result than Instruct-Pix2Pix. We also need to point out that Instruct-Pix2Pix (Brooks et al. (2023) relies on an instructive tuning-based diffusion model, which necessitates a rigorous and exten- sive training schedule. This stands stark con- trast to our method. Our paradigm is notably training-free, eschewing the need for such in- tensive data processing and model adjustment. sets our method apart but also underscores its efficiency where extensive training schedules are either impractical or undesirable.
Input Image
(ours)
Instruct-Pice
"Change the ked branches"
"Charge the photo Starry Night style painting"
Figure 16: Qualitative comparison with Instruct- Pix2Pix.
This fundamental difference in approach not only
J AMBIGUOUS INSTRUCTION ANALYSIS
To evaluate the robustness of DVP, we con- ducted an ablation study to determine the qual- ity of the translated image under ambiguous in- structions. The results in Fig. 17 show that despite the vagueness of the instructions, our approach is still capable of translating images with reasonably accurate results (e.g., when having "apples" as the only instruction, DVP able translate both ice-cream and burger to the apples). The validity of translated results can be primarily attributed the advanced ca- pabilities of GPT planning (see $3.2). GPT's understanding of nuanced instructions and its ability to generate coherent plans from incom- plete or unclear data played a pivotal role. Our approach thus not only excels in straightforward scenarios but also possesses commendable level of adaptability when faced with ambiguous or less-defined instructions. Such a feature valuable in real-world applications where clear and precise instructions may not always be available.
Input Image
DVP
AAplees"
"Car"
Figure 17: Ambiguous instruction analysis.
K COMPLEX INSTRUCTION ANALYSIS.
DVP harnesses the strong capabilities of the GPT its core planning mechanism. In Fig. 18] we demonstrate how DVP applies com- plex textual augmentations translate the im- age effectively. In this example, the instruction given as 'Swap the dog on the far right to the most left, and then swap the dog in the middle to the right" necessitating the planner to inter- pret and process the intricate/multiple textual descriptions "to the most left" and "on the far right." DVP shows a proficient capability in com- prehending the underlying textual descriptions and subsequently creating satisfying results. This illustration exemplifies how, by leveraging the advanced capabilities of GPT, our approach is also adept managing and handling complex arguments.
Input Image
DVP
"Swap dog the far right to the most left, and then swap the dog in middle to the right."
Figure 18:
Complex textual arguments.
19

---

Published as a conference paper at ICLR 2024
L ERROR ANALYSIS
In Fig. 19 we present an overview of the most notable failure cases and derive conclusions concern- ing their distinct patterns that result in less-than-optimal outputs. Evidently, our DVP encounters difficulties in differentiating and translating objects from backgrounds with challenging situations, such as poor photometric conditions (Fig. 19 bottom) and occluded objects (see Fig. [9 top). For instance, in the first row, poor lighting causes the translation to appear odd; in the second row, under unfavorable photometric conditions, both shape and shading are mistakenly identified as objects and utilized for subsequent translation; in the third row, the cat is obscured, making its translation into full tiger challenging; in the fourth row, due to the crowded nature of the lychees, their count varies after translation. The translation on these challenging situations remains a common limitations ob- served across the majority of concurrent diffusion models, Additional research in this domain is thus imperative. A viable approach involves the integration of amodal segmentation models (Zhang et al., 2019 Zhu et al. (017b) (i.e.. considering both the visible and occluded parts of the instance) to fully restore object masks prior to the reconstruction process. We highlight this as a future direction for addressing challenging scenarios.
M USER STUDY
In Table [ we design user study including "Quality," "Fidelity" and "Diversity," respec- tively. Specifically, we employ user study on Lik- ert scale (Likert (932), a well-established rating scale metric utilized for quantifying opinions, at-
Table 4: User study on controllability and user- friendliness.
Method Controllability User-friendliness
VISPROG
25.1 %
39.7%
DVP (ours)
74.9%
60.3%
titudes, or behavioral tendencies (Allen Sea-
man 2007 Bertram 2007: Boone Jr & Boone
2012 Norman 2010). This scale typically presents respondents with either a statement or a ques- tion. In case, the questions are measuring the overall harmony of the translated image (i.e., "Quality"), the preservation of the identity in the image (i.e., "Fidelity"), and variations between the original and translated images (i.e., "Diversity") rated by users from (worst) to 5 (best), Re- spondents then select the choice that most closely aligns with their own perspective or sentiment regarding the given statement question.
We further provide user study in Table focusing on the usability of explainable controllability discussed in $4.3 The results demonstrate that our approach facilitates user-friendly error correction and permits the intuitive surveillance of intermediate outputs.
N DISCUSSION
Novelty. Our DVP contributes on three distinet technical levels. Specifically, we first re-consider the design of classifier-fres freef guidance, and propose a conditional diffusion model which eschews the reliance on the sensitive manual control aaper-parameter w. Second, by decoupling the intricate concepts in feature spaces into simple symbols, enable context-free manipulation of contents via visual programming, which is both controllable and explainable. Third, our GPT-anchored pro- gramming framework serves as demonstrable testament the versatile transferability of Language Model Models, which can be extended seamlessly to a series of critical yet traditional tasks (e.g.. detection, segmentation, tracking tasks). We do believe these distinctive technical novelties yield invaluable insights within the community.
Limitation and Future Work. Despite DVP showcases superiority over state-of-the-at methods in both qualitative and quantitative aspects (see $4.2), it also comes with new challenges and un- veils some intriguing questions. For example, our approach grapples with obscured objects, at- tributed mainly to the segment module's limitation in exclusively processing the visible segment of an object, neglecting the occluded portion. This limitation consequently influences the diffu- sion process to generate base merely on the segmented part. In addition, in scenarios of photo- metric conditions, DVP and its counterparts falter in accurately aligning with the translated ob- ject(s). echoing challenges akin to the aforementioned issue (see $.. We posit that special- ized, fine- -raingr dataset tailored for a specific purpose (e.g., occluded items, photometric con- ditions) might be the future direction bolster the diffusion model's proficiency. In our study,
20

---

Published as a conference paper at ICLR 2024
integrated instance normalization with the null-text optimization, which is specifically design for text-guided diffusion model. This integration, however, is not directly applicable to a wider range of diffusion-basell image generation applications. Nonetheless, the substantial potential of instance normalization has not been fully explored or utilized yet. A promising direction to lever- age the power of instance normalization lies in fine-tuning the text-to-image task, which is split into a two-step procedure: image generation followed by image fine-tuning. Specifically, we can first generate some base images with any tex(2image model. Subsequently, ploy our approach with instance normalization to tune these images. For example in Fig. 20) we initially produce an image following comprehensive instructions and then proceed to fine- tune the images directly, eliminating the need for complete regeneration the image from scratch.
can naturally em-
This pipeline enhances the quality of the generated images, tailoring them to achieve the satisfying re- sults. Moreover, fine-tuning exist- ing images presents a more time and computationally efficient alternative compared to the regeneration from scratch each time. This effort also
Inpuu Instruction
Base Image
Fine-tuned Tmage
apple neating on wooden apple brightt droplets
Any diffusion its skin look fresh and appetizing"
"Change Smith."
Figure 20: Expand instance normalization to broader image generation pipeline.
has the potential to handle image variations in lighting, contrast, and other characteristics. Societal Impact. This work introduces DVP neuro-symbolic framework for image translation, showing robust image translation, strong in-context reasoning and straightforward controllability and explainability. On positive side, our framework reaches superior image translation performance qualitatively and quantitatively, and provide a user-centric design for the integration of future ad- vanced modules. DVP holds significant merit, particularly in applications pertinent to safety-criticn domains and industrial deployments. For potential negative social impact, our DVP struggles in handling obscured objects and photometric conditions, which are common limitations almost all concurrent diffusion models. Hence its utility should be further examined.
Ethics Concerns. The inherent design of the DVP, characterized by its modular structure, implies that it does not intrinsically contribute knowledge understanding the process of image trans- lation. This modular architecture indicates that DVP acts as a framework through which various processes are executed, rather than a mnowledge-contrbuutig concerns related to ethics or bias within the DVP system are predominantly relevant to its individual modules, rather than the system as a whole. It is worth noting that the modular nature of DVP of- fers a significant advantage in addressing these biases and ethical concerns. Since each component operates as discrete unit within the framework, modules that exhibit concerns. Once identified, these biased modules can be replaced with neutral alternatives. This adaptability crucial maintaining the integrity and effectiveness of the DVP, ensuring that it remains robust and fair tool in the realm of image translation.
entity its own right. Therefore, the
feasible to identify and isolate the individual
21

---

Published as a conference paper at ICLR 2024
Input Video
Ours (oil painting)
T
Input Video
Ours (oil painting)
1
Input Video
Ouis (oil painting)
Figure 12: Video translation results. Devoid of any dedicated temporal module, our approach demonstrates commendable performance outcomes when applied video data. D and O indicate artifacts and missing objects respectfully.
22

---

Published as a conference paper at ICLR 2024
VRS
There are four goldfish in tank with sand and plants
There is a cat that is laying down on a window sill
Anime girl holding a basketball in front of crowd of people
Scratch two teddy bears sitting next girl
A white jeep driving through desert with mounfain in background
Painting of a mountain with snow covered valley
There is golden humanoid robot that is standing in the street with gun
There is a phenix firebird that is flying over flowery branch
Cartoon robot with a rocket and spaceship in the background
There are five sunf lowers in a vase table
Dolphin riding on the water with a wave behind
There is German shepherd running in the grass
"Change the left/right fox to the arctic wolf"
Three foxes standing on rock front of a forest
Two foxes standing the right of wolf that is perched on rock in front of forest
Two foxes standing to the left of a wolf that is perched on rock front of forest
Figure 13: General-context dataset contains diverse image-text pairs (top three rows), and DVP presented images with targeted translation of the Rol (bottom two rows).
23

---

Published as a conference paper at ICLR 2024
Input Image
DVP (ours)
VQGAN-CLIP
Prompt2Prompt
VISPROG
"Change the toy husky to toy sheep"
"Change: the left person to robot"
"hannge the yellow rubber duck real chick"
Change the bell to donut"
"Change the right pigeons to eogles"
"Change the Bundt cake to pizza"
"Change the apples peaches"
Figure 14: More qualitative results of DVP. As seen, DVP shows high-fidelity image translation with context-free manipulation.
24

---

Published as a conference paper at ICLR 2024
"Change the kid to robot" / "Change the dog to cat then move it to the bottom"
Bad photometii condition
"Change the pears to apples with oil painting style" "Change the pears to gourds with oil painting style"
"Change the cat to the tiger then move it to the right of the dog"
Occluded objects
"Change the lychees to the Cherries then move cherry to the top of the apples"
Figure 19: Failure cases. Current DVP might get unsatisfied results when the input image is in poor photometric conditions or contains occluded objects.
25


Reviews:
[{"review_id": "6QokFmdT9c", "reviewer": "Reviewer_fWRk", "summary": "In this paper, the authors propose a new framework called Diffusion Visual Programmer (DVP) for image translation tasks. It follows the diagram that first identifies the instructed target region and then translates it into the targeted domain. In particular, the authors use instance normalization for context-flexible translation, avoiding adjusting guidance manually. Besides, the authors decouple the concepts in feature space into simple symbols and then deal with the symbols with visual programming. Qualitative and qualitative results show that DVP outperforms other state-of-the-art methods, providing more reliable, controllable, and interpretable image translation.", "questions": "- I have questions about the learning process of the 1×1 conv layer in equation (5). How is it exactly trained? And is it sensitive to the training sample size?\n- Will instance normalization also work in text-to-image tasks? It will be interesting to see if it could generate higher fidelity images with semantic meaning more aligned with the provided text prompts.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, the paper is well-organized and easy to follow. The figures and tables are informative.\n\n- The performance of the proposed method is promising. Figures 4, 6 clearly demonstrate the superiority of DVP.\n\n- The ablation study and system analysis are clear and informative, making it easy to see the effectiveness of different parts, such as instance normalization, and prompter.", "weaknesses": "- The proposed method is a systematic approach for image translation tasks incorporating different components. A potential drawback is its inference speed. It would be beneficial if the authors could compare inference speed with other image translation tasks.\n- The comparison with methods like SDEdit, Prompt2Prompt, and InstructPix2Pix is somehow unfair since they do not require an additional segmentation network.\n- The quantitative evaluation is only the proposed dataset, which contains fine-grained edit instructions. The effectiveness of DVP could be further proved by evaluating simple or even ambiguous instructions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "orOdJDGPTX", "reviewer": "Reviewer_TXue", "summary": "In this paper, the authors introduce the Diffusion Visual Programmer (DVP), a neuro-symbolic image translation framework that seamlessly combines a diffusion model with the GPT architecture. DVP enables transparent and controllable image translation processes, covering tasks like RoI identification, style transfer, and position manipulation. Extensive experiments showcase DVP's remarkable performance, surpassing existing methods. The success of DVP can be attributed to its condition-flexible translation, in-context reasoning, and systemic controllability. This research represents a significant advancement in harmonizing image translation with cognitive intelligence, with promising applications in various domains.", "questions": "While the paper showcases DVP's impressive performance, there is a need for further exploration of its adaptability to a range of image translation tasks. These tasks might include object replacement, color/texture/material editing, and style transfer. Currently, the paper's experiments encompass a holistic approach without distinct analysis of how DVP caters to varying task requirements.\n\nAdditionally, the paper's reliance on a relatively small self-collected dataset comprising 100 image-text pairs may present limitations in terms of the breadth and diversity of data for evaluation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Innovative Framework**: DVP introduces a novel neuro-symbolic image translation framework, seamlessly combining a diffusion model with the GPT architecture for various image processing tasks.\n\n2. **Remarkable Performance**: DVP outperforms existing methods, demonstrating high-quality image translation through extensive experiments.\n\n3. **Condition-Flexible Translation**: DVP achieves condition-flexible translation via instance normalization, enhancing content generation while reducing manual guidance sensitivity.", "weaknesses": "1. **Novelty**: The paper should address concerns about its novelty, as it resembles existing visual programming pipelines with the primary distinction being the choice of image editing tools. Clarification on how DVP significantly sets itself apart from previous work is crucial.\n\n2. **Lack of Comparative Analysis**: The paper's strength in performance could be further substantiated by a more thorough comparative analysis. A comparison with alternative approaches like instructive tuning-based diffusion models, such as instruct2pix, would provide a clearer perspective on its relative merits.\n\n3. **Limited Experimentation on Instance Normalization**: The paper introduces instance normalization as a valuable element of the DVP framework, which can have broader applications. However, there is a need for more extensive experimentation to validate the advantages of instance normalization in the context of general diffusion-based image generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IfVpWRCCs7", "reviewer": "Reviewer_DvDX", "summary": "This paper introduces a new method, named Diffusion Visual Programmer (DVP), for image translation. DVP is a neuro-symbolic framework that incorporates condition-flexible translation into an LLM planner. Specifically, the authors propose instance normalization to avoid manual guidance to achieve higher-quality content editing. The authors also prompt LLM to generate a sequence of programs with operations, which can be executed to plan the procedure, segment the object of interest, give the caption for the image, inpaint the masked image, and manipulate the sizes and positions of objects. DVP makes the process of image translation controllable and interpretable. The qualitative and quantitative results demonstrate a good performance of DVP.", "questions": "- The authors use a conv layer in instance normalization (Eq. 5). But there is no evidence of how the conv layer is trained or parameterized. Can authors give more details?\n- In Figure 7, We can see different plans derive different translated results. How do authors obtain different orders of operation sequences? How do authors decide which plan is the best for the final result?\n- In Fig. 8, are reconstructed results with automatic labels and human labels generated by null-text inversion? To my knowledge, null-text inversion produces quite similar reconstructed images to the original ones regardless of the input prompt. Can authors explain why using GPT-generated prompts derive better reconstructed results than human annotations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper integrates the condition-flexible diffusion model into a visual programming framework to improve image translation.\n- The paper proposes an efficient instance normalization to improve the quality of image editing.\n- GPT planner provides controllability and explainability for the image translation process.\n- The generated results are good with quantitative and qualitative evaluation.\n- The paper is well-written and easy to read.", "weaknesses": "- It seems the conditional-flexible diffusion model is the prompt-to-prompt method with proposed instance normalization. However, the authors only compare the results of using instance normalization and different guidance scales. It is also important to ablate the instance normalization in DVP to demonstrate instance normalization significantly outperforms a fixed scale when incorporating in-context visual programming.\n- The proposed method is an aggregation of different models planned by an LLM. However, how the components (e.g., mask2former, Repaint, BLIP, position manipulator) are aggregated is not clearly clarified. The authors may explain various translation procedures by providing more step-wise illustrations like Fig. 7. Besides, the authors may provide the prompt they use for the GPT planner.\n- Some important details are missing. For example, how the conv layer in instance normalization is parameterized? Is there any other difference between the conditional-flexible diffusion model and the prompt-to-prompt model, except for instance normalization? How to decide the order of operation sequences, automatically by GPT or manually by humans?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CHCGrPrhla", "reviewer": "Reviewer_EyuE", "summary": "The paper presents the Diffusion Visual Programmer (DVP), a new neuro-symbolic framework for image translation that combines a diffusion model with the GPT architecture, coordinating a sequence of visual programs for image editing tasks like RoI identification, style transfer, and positioning. DVP stands out due to its ability to use instance normalization for condition-flexible translation, focusing on textual prompts for quality content creation, and its capability to convert complex concepts in feature spaces into simpler symbolic representations, ensuring localized editing that retains image coherence. Additionally, DVP provides clear symbolic representations at every stage, enhancing user control and understanding, marking progress in aligning artificial image processes with human cognitive intelligence.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The Diffusion Visual Programmer (DVP) method, rooted in a neuro-symbolic perspective, integrates a condition-flexible diffusion model with the GPT framework, guiding a series of pre-existing computer vision models to produce the desired results. DVP's strength lies in its ability to accurately position translated objects and allow for various context-free manipulations. The advantages of this work include:\n1. Precision in Targeting: By prioritizing the identification of the Region of Interest (RoI), the system ensures accurate and relevant image translations.\n2. High-fidelity Translations: After identifying the target region, the system maintains high-quality translations into the desired domain.\n3. Spatial Awareness: DVP's capability to position translated objects based on spatial data enhances the translation's realism and context.\n4. Enhanced User Control: The system's step-by-step approach allows users to have better control over the translation process, ensuring results align closely with their intentions.", "weaknesses": "1. Dependency on Modules: DVP's systematic design implies that if one module falters, the entire system could be compromised. The robustness of such an interconnected system becomes a pertinent question. If one cog in the machine is faulty, can the system still deliver optimal results, or will it be critically hampered?\n2. Comparison with Other Techniques: Some results showcased by DVP can be achieved using established perception techniques like Segment Anything (SAM) or conditional control generation methods like ControlNet's point-to-point mode. A comparative discussion on the advantages of DVP over these techniques would provide clarity on DVP's unique value proposition.\n3. Textual Argument Handling: Even though DVP marries a condition-flexible diffusion model with the GPT architecture, its prowess in handling complex textual arguments requiring intricate reasoning remains questionable. The paper does not offer substantial experimental evidence to vouch for the enhanced capabilities of integrating GPT, leaving room for skepticism.\n4. Handling Complex Scenarios: DVP's recurrent challenge is its inability to adeptly manage intricate situations. As highlighted in the failure case study, the system falters in the face of occlusions, inadequate lighting conditions, and densely populated scenes. This limitation raises concerns about the system's applicability in real-world, diverse scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)