ChatEdit: Towards Multi-turn Interactive Facial Image Editing via Dialogue
Authors: Xing Cui
Zekun Li
Pei Pei Li
Yibo Hu
Hailin Shi
Chunshui Cao
Zhaofeng He
Venue: EMNLP 2023 Main (2023)
Keywords: Interactive image editing
Task-oriented dialogue

Abstract: This paper explores interactive facial image editing through dialogue and presents the ChatEdit benchmark dataset for evaluating image editing and conversation abilities in this context. ChatEdit is constructed from the CelebA-HQ dataset, incorporating annotated multi-turn dialogues corresponding to user editing requests on the images. The dataset is challenging, as it requires the system to dynamically track and edit images based on user requests, while generating appropriate natural language responses. To address these challenges, we propose a framework comprising a dialogue module for tracking user requests as well as generating responses, and an image editing module for editing images accordingly. Unlike previous approaches, our framework directly tracks the user request of the current turn from the entire dialogue history and edits the initial image instead of manipulating the output from the previous turn, mitigating error accumulation and attribute forgetting issues. Extensive experiments on the ChatEdit dataset demonstrate the superiority of our framework over previous methods and also improvement rooms, encouraging future research. We will release the code and data publicly to facilitate advancements in complex interactive facial image editing.

CHATEDIT: Towards Multi-turn Interactive Facial Image Editing via
Dialogue
Xing Cui
Zekun Li
Peipei Li
Yibo Hu
Hailin Shi
Chunshui Cao
Zhaofeng He
Beijing University of Posts and Telccommunication
UUniversity of California, Santa Barhara
$NIO 'WATRIX.AI
(cuixing, lipeipei, zhaofenghe j&bupt. edu. en
zekunl fecss ucsb.edu
(huy1bo871079699, hailinshi. www.wegmail..o.mmm.mmoo....w...w com
chunshui www.watrix..c..oo
<Useer> She should have long bengs.
<User>> Good idea. Plesse it
<Usse>> Lef me have look. Emm, how sbout trying black hair?
Heirstyle: bangs:
Havstyll: cangs:
Exprossion: smiling:
Heirstyle:
bengs
Expression: smilnng
has Mord har. She bunge. She has medumm mii.............................
-Systemm you have big smill??
<Sysemm you want to manipulate the hair color?
<yyseem> Ok, what alse do you want to try?
Figure I: Illustration of the multi-turn interactive facial image editing task. The system is required to track the user requests on the facial attributes (in the gray box), edit the image, and generate the response.
Abstract
This paper explores interactive facial image editing via dialogue and introduces the CHATE- DIT benchmark dataset for evaluating image editing and conversation abilities in this context. CHATEDET constructed from CelebA-HQ dataset, incorporating annotated multi-ture turn dia- logues corresponding to user edit requests on the images. The dataset challenging, as requires the system to dynamically track user requests, edit images, and generate appropri- ate responses. Accordingly, we propose three benchmark tasks: (i) user edit request tracking, (ii) image editing, and (iit) response generation. We present novel baseline framework that in- tegrates dialogue module for both tracking user requests and generating responses image editing module for image editing. Un- like previous approaches, our framework di- rectly tracks user edit requests from the entire dialogue history up to the current turn and mod- ifies the original image rather than adjusting the previous turn's 's output, thereby reducing accumulation and preventing attribute forget- fulness. Extensive experiments on the CHATE- DIT dataset underline framework's superior performance against prior models, while also highlighting potential room for further research. We will release the code and data publicly to facilitate advancements in complex interactive facial image editing* +
Input
Pale skin
Lipstick
Smiling
Black Hair
Figure 2: Comparison of previous repeated single- editing approaches and our proposed multi- turn editing approach. The cascaded errors in the single-turn approach lead to unintended changes in gen- der and eye makeup.
Introduction
an
With the rise of deep generative models such as GANs (Goodfellow et al., 2020; Karras et al., 2019; Li et al., 2023a) and DDPMs (Ho et al., 2020; Sohl- Dickstein al., 2015; Zhang et al., 2023), signif- icant progress has been achieved in instruction- based facial image editing (Xu et al., 2022; Patash- nik et al.. 2021; et al.. 2023b). However, an emerging scenario is multi-turn interactive edit- ing, allowing users to iteratively refine their editing instructions through system interaction (Sharma et al., 2018; Cheng et al., 2020; Kim et al., 2019; Jiang et al., 2021). Existing approaches (Zhou al., 2022: El-Nouby et al.. 2019; Kim et al., 2019) typ- ically treat multi-turn editing as a sequence of suc- cessive single-turn edits, leading to issues such as attribute forgetting and error accumulation, as depicted in Fig. (first line). Moreover, these tech-
Equal contribution.
Corresponding Author.
Our data and codes are available at //tp:::///tthll con/ com/cuixing cuixing1000 90876/ChatEdit

---

Table 1: Comparison of our CHATEDIT with existing works on image editing.
Method
Scene
Multi-tur Interaction
System Feedback
Dataset
Text
Attrihates
Transdditrr (Xu et 2022)
Facial image
CelebA IIQ
DaieCLIP (Wei et 2022)
Facial image
x
X
CelebA-11Q
StyleCIJP (Patachnik
2021)
image
CelebA-HQ
ChaiPainter (Sharma et ol., 2018)
Realistic image
MS COCO
dialoguc
TIGAN (Zhou et al., 2022)
Facial image
CelebA-IIQ
Talk-to- (Jiang at.l., 2021)
image
CelchA-dialog
user utterance
CHATEDIT (Ours)
Facial image
CHATÉDIT
dialogue
HairCLIP only considers huirstyle and hair color. 44 text deseriptions collected for hairstyle and 12 text descriptions for hair color.
niques don't fully harness interactivity and user experience. For example, (Sharma et al., 2018; Cheng et al., 2020; Kim et al., 2019) solely process user inputs without offering natural language feed- back, while others (Jiang et al., 2021) rely on rigid, hand-crafted response templates, limiting flexibil- ity and naturalness.
turns, thereby reducing error accumulation and tribute forgetting issucs. We perform extensive experiments with diverse settings to investigate the effectiveness of our proposed framework on the CHATEDIT dataset. The results suggest that our proposed framework is superior to the previously prevalent cascaded single-tur editing methods garding both image editing quality and response diversity.
To facilitate the research on multi-turn interac- tive facial image editing, we introduce a novel benchmark dataset named CHATEDIT. Sourced from the CelebA-HQ dataset (Karras al., 2018), CHATEDIT enhances a selected set of 12k images with annotated multi-turn dialogues that align with user edit requests for facial images. The annota- tions include user utterances, system responses, and the user's "belief state", which represents the user edit requests from the beginning of the dialogue to the current turn. Fig. illustrates that success CHATEDIT necessitates the system to track the user edit requests, edit images based on tracked requests, and provide natural language responses to engage with users. To evaluate the performance of multi-turn interactive editing, we define three tasks: (i) user edit request tracking, (ii) image edit- ing, and (iii) response generation. Correspondingly, we introduce comprehensive set of metrics that evaluate both response and editing quality.
To sum up, our contributions are three-fold: (1) We introduce the CHATEDIT benchmark dataset, which could serve as valuable resource for ad- vancing research in multi-turn interactive facial im- age editing. (2) We propose a novel framework that seamlessly combines task-oriented dialogue mod- ule and an image editing module. This framework effectively tracks user requests, performs image editing. and generates system responses. Impor- tantly, it addresses the issues of attribute forgetting and error accumulation prevalent in previous meth- ods. (3) Through qualitative and quamtitative evalu- ations on the CHATEDIT dataset, demonstrate the superiority of our proposed multi-turn editing framework over previous cascaded single-turn edit- ing methods. We believe these results not only highlight the strengths of our proposed approach but also spur further exploration in this field.
Based on our benchmark dataset, we propose baseline framework for multi-turn interactive im- age editing. Our framework seamlessly integrates language dialogue module and a visual image edit- ing module. Specifically, we employ an end-to-end task-oriented dialogue (TOD) model to extract the user's image edit requests and generate appropriate responses based on the current conversation con- text. These tracked user requests are transformed into a text prompt to guide the baxt- image editing model, StyleCLIP (Patashnik et al., 2021), in performing the desired manipulations on the in- put raw image. As illustrated in Fig. 2's second line, our benchmark dataset and the proposed framework enable direct editing on the input raw image instead cascaded modifications to images from previous
2 Related Work
Facial Image Editing Traditional image edit- ing techniques have focused on modifying specific given attributes such as age (Li al., 2020a, 2019), hairstyle (Wei et al., 2022), or other predefined at- tributes (Zhou et al., 2022; Li et al., 2020b; Zhang et al., 2022). In recent years, with the develop- ment of pre-trained vision-language models (Rad- ford et al., 2021; Li et al., 2022a; Wang et al., 2023), there has been growing interest in human- computer interaction scenarios (Kottur et al., 2022: Jiang et al., 2021). This leads researchers explore interactive image editing, where users can dynam- ically adjust their editing requests through inter- action with the system (Kim et al., 2019; Lachmy

---

Table 2: Illustration of editable attributes in CHATE-
Table 3: Statistics of dialogues in CHATEDIT dataset.
DIT, which categorized into four groups (slots).
Total # dialogues
12,000
Slot
Attribute
Total # utterances
96,174
Avg If turns per dialogue
4.0
Expression
smillig, no smiling, angry. sad
Avg # utterances per dialogue
8.0
Hair color
brown hair, blond hair, black hair, gray hair
Avg # words per user turns
11.6
Hair
receding hairline, sideburns, bangs, no bangs, mustachee, goatee, beard
Avg # words per system turns
11.3
Avg # attributes mentioned per dialogue
6.3
Makeup
no makeup, heavy makeup, lipstick, bushy eye- brows, rosy cheeks, pale skin
ing and consulting, neglecting the interactive image editing scenario.
et al., 2022; Zhou et al., 2022: Jiang et al., 2021). For instance, TiGAN (Zhou et al., 2022) generates images iteratively based on successive editing steps during conversation.
3 CHATEDIT Benchmark Dataset
CHATEDIT is designed simulate scenario where a user interacts with a system to manipu- late a facial image. Each sample in the CHATEDIT dataset comprises a facial image and a dialogue be- tween the user and the system regarding the editing of the image. Notably, each turn of the dialogue is annotated with the user belief state, representing the user requests that guide both response gener- ation and image editing. In this section, we will first introduce how to construct the dataset and its statistics. Then, we introduce benchmark tasks CHATEDIT and corresponding evaluation metrics.
However, these methods merely "listen" to user requests without generating system responses, con- straining their interaction capability. A recent work called Talk-to-Edit (Jiang et al., 2021) attempts to address this by introducing a rule-based method to generate system response. However, this approach lacks flexibility and struggles with unforeseen sce- narios that are not predefined. Furthermore, these methods treat multi-turn editing as sequence of individual single-turn edits, resulting in issues such error accumulation and attribute forgetting as the number of interactions increases.
3.1 CHATEDIT Dataset Construction
Task-oriented Dialogue There are two primary approaches to TOD. Traditional systems adopt a pipelined approach comprising four modules. Firstly, the natural language understanding (NLU) module (Abro et al., 2022) converts user requests into semantic slots, domain information, and user intention. Secondly, the dialogue state tracking (DST) module (Wu et al., 2019; Le et al., 2019; Lin et al., 2021; Heck et al., 2023) extracts the dialogue state, which records user requests in the form of slot-value pairs. The dialogue policy learn- ing (POL) module (Chen et al.. 2017: Geishauser al., 2022) determines the next action of the dia- logue agent based on the dialogue state. Finally, the natural language generation (NLG) module (Elder et al., 2020) generates the system response. In more recent times, there has been a shift towards end-to- end task-oriented dialogue systems (Hosseini-Asl et al., 2020; Wang et al., 2022; Le et al., 2020; Zeng et al., 2023). For example, PPTOD (Su et al., 2022) employs multi-task training that simultanc- ously processes all sub-tasks. Currently, there have been approaches that utilize Large Language Mod- els (LLMs)) like ChatGPT for task-oriented dia- logues (Li et al., 2023c). However, existing TOD systems primarily focus on scenarios such as book-
Facial Image Data We construct the CHATEDIT dataset on top of the CelebA-HO (Karras et al., 2018), which is high-resolution facial image dataset with 30k images. It provides binary annota- tions of 40 facial attributes. We select 17 attributes from CelebA-HQ and add another four frequently- used attributes "sad", "angry", "no smiling", and "no bangs", resulting in a total of 21 editable at- tributes in the CHATEDIT dataset. As shown Table 2, these editable attributes are categorized into four groups. We select 12k images from the CelebA-HQ and utilize the annotation in CelebA- HQ build caption for the image. An example is presented in Fig. in Appendix.
Dialogue Annotation During each turn of the dialogue, the user expresses their editing requests using natural language. is essential for the system to detect these requests as belief states and map them to appropriate responses. Consequently, we annotate three types of data for cach turn in the dialogue: (1) the user utterance, (2) the user belief state (user requests), and (3) the system response.
We here adopt terminology commonly used in lask-oriented dialogue research to introduce the an- notations. Each group in Table is considered a

---

8000
Frequency
5080
1.11%
1.88%
0-4
4000
5-8
2.29%
9-12
1000
2.68%
13-16
enggyy bangs
black Hair
blond Hair
bushy yobrown
brown hair
gostbs
gray hair
haavy makoup muelrche bangs no bourd no nakeup amiling paia skin
meeding hairinn raay choeks
aad sideburns smiting lipelick
4.01%
17-20
9.50%
21-24
Figure 3: Analysis of CHATEDIT dataset. Left: Distribution of different attributes. Right:
Distribution of
occurrence frequency of various attribute combinations.
User-1
System-1
System-2
System-3
editable attributes (excluding the original attributes of the raw image) and expressing them as slot-value pairs. Secondly, we select an utterance that effec- tively expresses this particular request. Then, based on a predefined policy, we determine the appropri- ate system action and choose a candidate system response. Specifically, the predefined policy is a set of rules that govern the dialogue flow to ensure it aligns logically with the user's s instructions and system functions. For instance, if user requests change in hair color blond, the system's 's pre- defined policy would prevent it from generating redundant or illogical suggestion like "Do you want to dye your hair blond?". Finally, the multi- turn dialogue can be constructed by repeating the above steps.
Excression
Expression
Next
Express on
Next
rolor
Reeg-io
Hai color
Ricaucsa
Hai color
Reauest
Har
Har
Her
Sugges:
Suggest
Sigget
Figure 4: Illustration of the dialogue flow for the first 3 turns in the CHATEDIT dataset. User-n and System- n represent the nith user and system turn, respectively.
slot, and each editable attribute represents possi- ble slot value for its corresponding slot. As result, the user request in each turn of the dialogue is rep- resented as slot-value pairs. For example, user request could be "expression: smiling, hairstyle: bangs, hair color: black hair."
To reduce human efforts in annotating the dia- logues, we leverage a two-phase pipeline (Karras et al., 2018) consisting of simulation phase (Li et al., 2022b) and paraphrase phase. During the simulation phase, begin by preparing collec- tion of varied human-written utterances that de- scribe user requests for each editable attribute and system responses for each system action. The sup- ported system actions include Next: general queries on what to edit in the next turn, Request: request whether to edit an attribute, Suggest: suggestions to edit towards specific attribute value. We then utilize ChatGPT (Ouyang et al., 2022) to generate more diverse utterances by combining task instruc- tion and few human-writted example utterances the prompt as shown below (Li et al., 2021).
Following the simulation of multi-turn dialogues, manual review is conducted to ensure their qual- ity. Human annotators are tasked with reviewing the dialogues ensure they are following the pre- defined dialogue flow logic. Additionally, they are asked to refine the expressions to enhance versity and naturalness if necessary. Fortunately, this process of checking and revising the dialogues is less labor-intensive compared to annotating the dialogues from scratch, resulting in significantly reduced annotation efforts.
3.2 CHATEDIT Dataset Statistics
The constructed CHATEDIT dataset comprises 12k examples, with each example consisting of a facial image equipped with corresponding caption and annotated multi-turn dialogue. We divide the CHATEDIT dataset into training, validation, and testing sets with 10k, 1k, and 1k examples, respec- tively. To offer deeper insights into the dataset, we present comprehensive statistics in this section.
Write diverse sentences to express the given image editing require-
Requirements: smillng
Sentence Can you edit the image show him with smile?
Sentence 2: would be great you could add smnle his fave.
Sentence
see
image
big smile on his face.
Sentence Ne
Analyzing Dialogues Our CHATÉDIT dataset comprises a total of 12k dialogues, consisting of approximately 96k utterances. The statistical de- tails of the dataset are presented in Table 3. More-
Next, we construct the dialogue by following a series of steps. Firstly, we determine the user request of the current turn by randomly selecting

---

over, we provide visual representation of the dia- logue flow in Fig. 4. Each block in the figure cor- responds to specific turn labeled as Start, User-n, or System-n, where n represents the corresponding turn index. For user turns, there are four differ- ent kinds of blocks, each representing an attribute group slot. Additionally, there are three types of system turns, each representing system action. The width of each block in the visualization indi- cates its occurrence frequency, while the connectiv- ity between blocks signifies their co-occurrences the dialog flow. Notably, the dataset exhibits a balanced distribution of different dialog flows, indicating a high level of diversity.
et al., 2015), where Distinct-n represents the ratio of distinct n-grams in responses.
Image Editing Editing an image according to the user request is primary objective of the CHATE- DIT system. Given user requests that consist of multiple attributes, robust system should be capa- ble of simultaneously manipulating these attributes while maintaining high quality. We evaluate the im- age editing performance from two perspectives: (I) relevance: whether the requested attributes are ac- curately edited; and (2) quality: whether the edited image is realistic and natural.
We measure the editing relevance of each at- tribute via the cosine similarity of the attribute and the edited image. To evaluate the relevance of mul- tiple attributes, Relevance (AvgRel) reflects the average editing rel- evance on all requested attributes, where the rel- evance is measured with the cosine similarity be- tween the edited image and the requested attribute:
Analyzing Editing Attributes Regarding the an- notated user requests, we provide insights into the frequency of each attribute (left) and the occur- rence frequency of different attribute combinations (right) in Fig. 3. The figure illustrates that the ma- jority of attributes are mentioned over 500 times, while more than 78% of the attribute combinations occur less than 5 times. These findings highlight the diversity present in our dataset.
report two metrics: Average
Avg Rel = avg {COSCLIP (1,t)eeT-
(1)
COSCLIP represents the cosine similarity function. T represents the set of all editing attributes.
3.3 CHATEDIT Evaluation
We also report Minimum Relevance (MinRel), which reflects the worst relevance among the re- quested attribute:
To evaluate the system's ability to detect re- quests, generate the appropriate system response. and edit the image according to user requests, we propose three benchmark tasks.
MinRel = min (COscLIP (I, (1,t))ee)
(2)
As for the measure of image quality, utilize FID (Heusel et al., 2017) and LPIPS (Zhang et al., 2018) metries, which calculate the statistical simi- larity between the edited images and the originals.
User Request Tracking Similar to the dialogue belief state tracking task in TOD, introduce the user request tracking task to evaluate the system's ability in detecting the user requests. The user re- quests are represented as slot-value pairs, where each slot denotes a category (group) of attributes and the value indicates specific attribute value. The input for this task is the dialogue history from the beginning to the current turn. The performance is evaluated with Joint Accuracy, which is the ratio of dialogue turns whose slots are predicted completely accurately, i.e., all the slot and slot val- ues are predicted correctly.
Method
In this section, we introduce our proposed frame- work for CHATEDIT. As illustrated in Fig. 5, the system consists of a task-oriented dialogue module and an image editing module. We adopt a pre- trained language model-based TOD model as the unified dialogue module for both User Request Tracking and response generation. Specifically, it takes the dialogue context prepended with differ- ent prompts as input and outputs the tracked user requests and corresponding system response. As for the image editing module, text-based image editing model StyleCLIP (Patashnik al., 2021) is utilized, which receives the user requests tracked by the dialogue model as input and edits the input image accordingly.
Response Generation Another objective of the dialogue module is to generate fluent, reasonable, and diverse responses to interact with the users. We use BLEU (Papineni al., 2002) to evaluate the fluency of generated responses with respect to reference responses. To evaluate the diversity of generated responses, we employ Distinct-1, (Li

---

franslate diatague to dialogue repoons. in phatt a women Sie nse bland hav She dbesitt have bongr. She has mediun aminng She wearig how loal amilgg <Systeno Bostik.. making bange longe??
System Response
Dinlogue
+Sydlerc Ze you want manbbulbte she now
give
Iranslate distogue to dialogue state, <ibwr wart to see how she wil look winoot smilig. <Systent Gk i's danw How about moving the honge?? Plaape
User Requests
Image Editing Module
Haissyfo Cango
Dialogue Module
Image Editing Module
Ideenity Loss
Trankbrerr Lagarh
Teankoen Loga
StyeGAN
Jou
Freeiiy
Jost
Frieday
Faaneyyl dange Expresation ariling
Zauga and do smiknn
Dalogue
Figure 5: Illustration of the proposed framework. The dialogue understanding module is utilized to track user requests and generate system responses. Then, the tracked user requests are fed to the image editing module to guide the image manipulation.
4.1 Dialogue Module
4.2 Image Editing Module
In this paper, we employ text-driven image edit- ing method inspired by StyleCLIP (Patashnik al., 2021). This approach combines the generative ca- pabilities of StyleGAN (Karras et al., 2019) with the joint vision-language representation learned by CLIP (Radford et al., 2021). To generate the ma- nipulated image based on user requests, we utilize CLIP- based loss, which optimizes the latent code of the input image to align with the directions in- ferred by the descriptive text of the user requests.
The primary goal of the dialogue module is to track user requests and generate natural language feed- back as responses. Following (Su et al., 2022), we formulate the two tasks as text generation and adopt a unified pre-trained language model T5 (Raf- fel et al., 2020) for both two tasks. Specifically, we prepend a task-specific prompt to the dialogue con- text to serve the dialogue language model's input, and the model trained to output the task-specific output. We use "translate dialogue to dialogue state" and "translate dialogue to dialogue response" as prompts for the User Request Tracking task and the response generation task, respectively. As the facial attribute values of the initial image can pro- vide valuable context to enhance the generation of reasonable conversations, age caption of the initial raw image into the input for the response generation task. Finally, the dia- logue model is trained to generate user requests and system responses in a multitask learning approach.
As the tracked user requests are stored in the slot-value pair format, we first construct them into the descriptive text prompt / with templates. Then, we manipulate the image by optimizing the latent code. Specifically, the images are projected to the latent codes and manipulated in the W+ space, which is extended from the W space proposed in StyleGAN (Karras et al., 2019). For a StyleGAN with 18 layers, W+ space is defined by the cascade of 18 different vectors [wi,.., w1s], Wy E W.
incorporate the im-
As shown in Fig. 5, the input image is first inverted by fixed StyleGAN inversion model e4 (Tov al.. 2021) to obtain the latent code of the input image w. E W+. Then, the latent code w E W+ is initialized as W.. w learnable and will be optimized towards the latent code of the edited image. Denote the optimization result of w as w*, w" can be viewed as the approximation of the latent code of the edited image. Then, taking w* as input, image via pretrained StyleGAN. In order to opti- mize w, first fed to the pretrained StyleGAN to obtain its corresponding image. Thereby, the supervisory information will be delivered via the image. Specifically, to force w to be consistent with
We use the T5 (Raffel et al., 2020) model and initialize it with the PPTOD (Su et al., 2022) check- point that has been pre-trained on a diverse set of dialogue corpus and thus equip with primary TOD task completion skills. Given task-specific prompt 2t. dialogue history x, and larget output y. The dia- logue model is trained with maximum likelihood objective and the loss function defined as:
can generate the desired edited
Le Σ log (yilvest 2),
where ④ is the model parameters.

---

Table 4: Quantitative results of the dialogue module user request tracking and response generation.
Table 5: Quantitative results of the image editing performance. Input represents the input of the image editing module. USR, Dial, and USR-T stand for ora- cle user requests, oracle dialogue, and the user requests tracked by the dialogue module respectively. All experi- ments utilize StyleCIIP to manipulate images.
User Request Tracking
Response Goneration
Model
Juint Acef
RUF
Di
Distint-21
T.xxl
88. 2112
0.503-0:025
0006=00008
Thus
00.065.0.11
0.108=0.005 199
0.872:0.002
T5ug
88,953-0
16.064-0.504
0.11--0.110
0.84
Editing Mode
Enput
Image Editing
FID
LPIPS
Mintel
AvgRel
the user's descriptive text, the CLIP loss (Radford et al., 2021) is utilized:
Single-tm
USR
41.11
0492600022
0.730=0010
0,752=0.010
Milti um
USR
00.118 10,16
6.619 0.001
0,754, 00099
0,773 0.609
Malti
Dial
42.11310,110
0.47710000
0.241
0.00
0.70110:000
Malii-um (Oun)
USR-T
40 536.0.66000
0,44+:00000
0.753=0000
0,773=00.10
CCLP - DCLIP (G (w),t) .t)
(4)
where G is pretrained StyleGAN generator that maps latent code into an image. DCLIP measures the cosine distance between the CLIP embedding of the image and the text. Then, L2 loss is utilized for preserving the similarity between the input image and the edited image in the W+ space:
As for the performance regarding the response generation, an improvement can be seen in both the BLEU score and the Distinct- 1,2 score are ob- served as the model size increases, demonstrating that the larger model can generate responses with better quality and diversity.
L = | wall2-
(5)
Image Editing Module The weight of \1, and \ID are 0.008 and 0.005, respectively, The editing step is 300, and the Adam optimizer is used with a learning rate of 0. 1. For fairness, all experiments utilize StyleCLIP to manipulate images.
Moreover, LID is used to preserve the identity:
LID = (R (G (wx)). R(G(w))),
where a pretrained face recognition net- work (Deng et al., 2019), (.-) calculates the cosine similarity. The overall optimization objective is:
To evaluate the effectiveness of our proposed multiple-tum editing approach, we first compare it with single-turn image editing. To eliminate the im- pact of misidentification of user edit requirements, the oracle user requests are utilized as the input of the image editing module. As shown in the first two lines in Table 5, significant improvements can be observed in all four metrics, indicating that our method successfully mitigates the error accumu- lation problem and achieves higher image quality. Moreover, we find that the MinRel score of the single-turn method considerably lower than that of the multi-turn method. This expected since the single-turn editing approach tends to forget some edited attributes from earlier turns, resulting in sub- optimal performance in the poorest attribute.
"" = arg minCcuP + A1.22 + AunLin. (7) weW+
5 Experiment
5.1 Quantitative Evaluation
Dialogue Module We initialize our dialogue model with diffcrent sizes of pretrained PP- TOD checkpoints PPTOD PPTOD and PaTOD respectively. We further fine-tune the model leveraging the Adam optimizer with learn- ing rate of 5e-5 and a batch size of 64. We utilize multi-task training strategy in which the User Re- quest Tracking task and the Response Generation task are trained simultaneously.
The last line in Table 5 presents the results of image editing using our proposed pipeline, where the image editing module edits the image based on the user requests tracked by our dialogue module. Notably, the performance achieved in this setting is comparable to the results when oracle user requests are taken as input. This finding highlights the effec- tiveness of our pipeline, particularly in real-world applications where obtaining perfect oracle user requests may not be feasible.
As shown in Table 4, we evaluate the dialogue module on the user request tracking and response generation task. All three models achieve joint accuracy of over 88%, indicating their effective- ness. We also compare our model with Chat- GPT. Specifically, we used the prompt proposed by (Heck al., 2023) leverage ChatGPT for User Request Tacking, which is up-to-date and has achieved impressive results on the MultiWOZ. dataset. Our method achieves a Joint Accuracy of 88. 88.86%, outperforming ChatGPT's 76.26%
Comparison with Talk-to-Edit We conduct a quantitative comparison between our method and

---

User: The person oan smal vary mapaay.
Ober. lecold like world
User: Can you adk ploto give gray
Osar leart by- asproosion
User: Can Kulp bargs
Usar. The hai anouh!
Talk-to-Edit
Talk-to-EEdt
Irput
Sysdem Winal ste peraae you
Syanem: ON, the'ss ersuga??
Systum: Finaled. oo
Ispat
Systemr: Are lee carip an propee
Spatdm Dingo wise orte?
Spodam: Poistaet goou snough??
sumo
Ours
Iiput
System Okay WS
System, Done, Agri kangs?
System eugo.
Ieput
System: Ther' Whe wht you
Syctem Ol Proskdd what
System Okay, aono. you wast outtingg
Figure 6: Visualization of the comparison between method with Talk-to-Edit. Left shows the attribute for- getting problem of Falk-to-Edit, where the smillig attribute is gradually diluted. Right shows the error accumulation problem. Besides, our method generate instructive responses by giving suggestions (highlighted in yellow).
Table 6: Quantitative comparison the image edit- ing performance between ours and Talk-to-Fdii
70%
00%
56 00.00%
Model
I.PIS
AsgRel 1
45%
30 10.00%
30%
Talk-to-Edit
Ours
132,555
98. 98.760
0.644
0.439
0.731
0,770
0,739
0,776
15%
12.00%
4.00%
12.00%
16.00%
Monipuatioo
wereiinngeee
Engagingness
the representative single-turn approach Talk-to- Edit on 100 randomly selected samples. As shown in Table 6, our framework achieves better image quality with lower FID and LPIPS scores, high- lighting reduced error accumulation. Moreover, method excels in MinRel and AvgRel metrics, underscoring its better alignment with user requests and effective mitigation of attribute forgetting.
Figure 7: Human evaluation of ours Talk-to-Edit on one axis on image editing: Manipulation and three axes on response generation: Humanness, Interesting- ness, and Engagingness.
present examples of manipulation results compar- ing our method with Talk-to-Edit in Fig. 6, show- casing the high image editing quality and response generation diversity achieved by method. More visualization results can be found in Fig. 14, 15, and 16 the Appendix.
Tracked user request vs Dialogue To better un- derstand the essentials of our introduced dialogue module, we experimented with using the raw dia- logue context instead of the extracted user requests as input for the image editing module. Table reveals that while the MinRel and AvgRel scores from this ablation study remain competitive with our approach, the FID and LPIPS scores drop no- tably. This discrepancy can be attributed to the presence of noisy information in the dialogue con- text. These results underscore the importance of accurately extracting user requests from dialogues through our proposed dialogue module. Further visual illustrations can be found in Fig. 13 in the Appendix.
5.3 Human Evaluation
To more comprehensively assess our method's re- sponse generation and image editing capabilities compared with Talk-to-Edii (Jiang et al., 2021), we conduct human evaluations. We choose 20 random images and initiate multi-ture interactive facial im- age editing dialogues using both method and Talk-to-Edie Five English-flue filuntt graduate students participate this assessment, conducting pairwise comparisons between the two methods. The eval- uations focus on one aspect of the manipulated image: manipulation, and three aspects of the gen- erated response: humanness, interestingness, and engagingness. The final answer for each question is determined by majority voting.
5.2 Qualitative Evaluation
In Fig. compare our multi-turn approach with the single-turn approach, illustrating the attribute forgetting and error accumulation problems of the single-turn approach. Additional visualization re- sults are shown in Fig. in the Appendix. We also
Fig. visualizes the evaluation results, under- scoring the superiority of our method. Specifically,
for image editing, our method effectively gener- ates images of high quality, taking advantage

---

the extracted concise user requests guide the im- age editing process. For response generation, our method generates more human-like, interesting and engaging responses, improving the interactivity of the system. By contrast, Talk-to-Edit relies on a rulc-based approach to generate template responses and employs cascaded single-turn image editing approach, limiting its performance.
Ethics Statement
is important to clarify that the facial images used in the CHATEDIT dataset are selected from CelebA- HQ (Karras et al., 2018), which is a dataset derived from CelebA (Liu et al., 2015). CelebA consists of images collected from the internet and is pub- licly available for research purposes only. The images in CelebA-HQ have undergone additional post post-processing. The dialogues the CHATEDIT dataset do not contain sensitive or private informa- tion. The dataset has been carefully curated to en- sure the privacy and confidentiality of individuals. Furthermore, participants involved in the manual paraphrase and human evaluation processes were compensated with reasonable wages.
6 Conclusion
This paper introduces the CHATEDIT benchmark dataset, which we believe could facilitate the re- search on multi-turn interactive facial image edit- ing. The dataset poses significant challenges as requires systems to accurately track user requests from dialogues, perform image editing based on these requests, and generate appropriate responses. We propose baseline framework that seamlessly combines task-oriented dialogue module and an image editing module. The introduction of the task- oriented dialogue not only enables interaction with users but also extracts concise user requests from the dialogue context to direct the image editing, avoiding the attribute forgetting and error accumu- lation issues in previous single-tury methods. The empirical results highlight the efficiency of our ap- proach and the potential for further advancements in this exciting research area.
Acknowledgement
This work is supported by Major Technology In- novation Program of Hangzhou, China (Grant 2022AIZD0154), National Natural Science Foun- dation of China (Grant No. 62306041, No. U21B2045), Beijing Nova Program (Grant No. Z211 10000212 106, 20230484488), and Na- tional Key R&D Program of China (Grant No. 2022YFF1202400..
References
Waheed Ahmed Abro, Annalena Aicher, Niklas Rach, Stefan Ultes, Wolfgang Minker, and Guilin Qi. 2022. Natural language understanding for argumentative dialogue systems in the opinion building domain. KBS.
Limitations
Our work is the first benchmark dataset explore multi-turn interactive facial image editing via di- alogue and establishes baseline performance for a variety of scenarios. However, there is room for improvement in the following aspects: 1) In the dataset construction, we consider 21 attributes editable attributes. However, there are out-of- domain attributes users might want to manipulate. In this case, the dialogue understanding may ne- glect the user requests or generate extra halluci- nations. 2) The proposed baseline model has two stages, which leverages the powerful capabilities of existing models via lightweight fine-tuning. How- ever, both the dialogue understanding module and the image editing module limit the quality of the manipulated image. This issue might be alleviated by training the whole model end-to-end, which will be included in our future research. In addition, other issues, such as how to construct a more gener- alized and robust facial image editing model, also require further exploration.
Lu Chen, Xiang Zhou, Cheng Chang, Runzhe Yang. and Kai Yu. 2017. Agent-aware dropout dqn for safe and efficient on-line dialogue policy learning. In EMNLP.
Yu Cheng, Zhe Gan, Yitong Li, Jingjin Liu, and Jian- feng Gao. 2020. Sequential attention gan for interac- tive image editing. ACM MM.
Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. Areface: Additive angular margin loss for deep face recognition. In CVPR.
Alaacldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W Taylor. 2019. Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction. In /CCV.
Henry Elder, Alexander O' O'Connor, and Jennifer Foster. 2020. How to make neural natural language genera- tion reliable as templates task-oriented dialogue. In EMNLP.

---

Joseph L Fleiss and Jacob Cohen. 1973. The equiva- lence of weighted kappa and the intraclass correlation coefficient as measures of reliability. EPM.
Hung Le, Doyen Sahoo, Chenghao Liu, Nancy Chen, and Steven CH Hoi. 2020. Uniconv: A unified con- versational neural architecture for multi-domain task- oriented dialogues. In EMNLP.
Christian Geishauser, Carel van Niekerk, Hsien-Chin Lin, Nurul Lubis, Michael Heck. Shutong Feng, and Milica Gasic. 2022. Dynamic dialogue policy for continual reinforcement learning. In COLING.
Hung Le, Richard Socher, and Steven CH Hoi. 2019. Non-autoreerressive dialog state tracking. In ICLR.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. a10.:3550.00355.
Jan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. CACM.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a. Blip:
Michael Heck, Nurul Lubis, Benjamin Ruppik, Renato Vukovic, Shutong Feng. Christian Geishauser, Hsien- Chin Lin, Carel van Niekerk, and Milica Gašié. 2023. Chatgpt for zero-shot dialogue state tracking: A solu- tion or an opportunity? arXiv:2306.0133 306.0186.
Bootstrapping language-inage pre- training for unified vision-language understanding and generation. In ICMI.
Peipei Li. Yibo Hu, Ran He, and Zhenan Sun. 2019. Global and local consistent wavelet-domain age syn- thesis. TIFS.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurlPS.
Peipei Li. Huaibo Huang. Yibo Hu, Xiang Wu, Ran He, and Zhenan Sun. 2020a. Hierarchical face aging through disentangled latent characteristies. In ECCV.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models. NeurIPS.
Peipei Li, Xiyan Liu, Jizhou Huang, Deguo Xia, Jianzhong Yang, and Zhen Lu. 2023a. Progressive generation of 3d point clouds with hierarchical con- sistency. PR.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A sim- ple language model for task-oriented dialogue. In NeurlPS.
Peipei Li, Yinglu Liu, Hailin Shi, Xiang Wu, Yibo Hu, Ran He, and Zhenan Sun. 2020b. Dual-structure disentangling variational generation for data-limited face parsing. In ACM MM.
Yuming Jiang. Ziqi Huang, Xingang Pan, Chen Change Loy, and Ziwei Liu. 2021. Tolk-to-edit: Fine-grained facial editing via dialog. In ICCV.
Peipei Li, Rui Wang, Huaibo Huang, Ran He, and Zhaofeng He. 2023b. Pluralistic aging diffusion toencoder. In ICCV.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehti- nen. 2018. Progressive growing of gans for improved quality, stability, and variation. In ICLR.
Zekun Li, Wenhu Chen, Shiyang Li, Hong Wang. Jing Qian, and Xifeng Yan. 2022b. Control- lable dialogue simulation with in-context learning. 22X.::2210.04185.
Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative ad- versarial networks. In CVPR.
Tero Karras, Samuli Laine, Miia Aittala, Janne Hell- sten, Jaakko Lehtinen, and Timo Aila. 2020. Analyz- ing and improving the image quality of stylegan. In CVPR.
Zekun Li. Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. 2023c. Guiding large language models via directional stimulus prompting. arXiv preprint 15X.
Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Mareus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh, 2019. Codraw: Col- laborative drawing as a testhed for grounded goal- driven communication. In ACI.
Zekun Li. Hong Wang, Alon Albalak, Yingrui Yang. Jing Qian. Shiyang Li, and Xifeng Yan. 2021. Mak- ing something out of nothing: Building robust task- oriented dialogue systems from scratch. Proceedings of Alexa Prize TaskBot.
Satwik Kottur, Seungwhan Moon, Aram H Markosyan, Hardik Shah, Babak Damavandi, and Alborz Gerami- fard. 2022. Tell your story: Task-oriented dialogs for interactive content creation. arXiv:2211.03942
Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul A Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, et al. 2021. Zero-shot dialogue state tracking via cross- task transfer. In EMNLP.
Royi Lachmy, Valentina Pyatkin, Avshalom Manevich, and Reut Tsarfaty. 2022. Draw me flower: Process- ing and grounding abstraction in natural language. TACL.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep learning face attributes in the wild. ICCV.

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In NeurIPS.
Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Zhentao Tan, Lu Yuan, Weiming Zhang, and Nenghai Yu. 2022. Hairclip: Design your hair by text and reference image. In CVPR.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. ACL.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In ACL.
Yanbo Xu, Yueqin Yin, Liming Jiang, Qianyi Wu, Chengyao Zheng, Chen Change Loy, Bo Dai, and Wayne Wu. 2022. Transeditor: transformer-base dual-space gan for highly controllable facial editing. In CVPR.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV.
Weihao Zeng, Keqing He, Yejie Wang, Chen Zeng, Jin- gang Wang. Yunsen Xian, and Weiran Xu. 2023. Fu- turetod: Teaching future knowledge to pre-trained language model for task-oriented dialogue. In ACL.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In JCML.
Richard Zhang, Phillip Isola. Alexei A Efros, Eli Shecht- man, and Oliver Wang. 2018. The unreasonable ef- fectiveness of deep features as perceptual metric. In CVPR.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. JMLR.
Zicheng Zhang, Bonan Li, Xuecheng Nie, Congying Han, Tiande Guo, and Luoqi Liu. 2023. Towards consistent video editing with text-to-image diffusion models. In NeurlPS.
Zicheng Zhang, Yinglu Liu, Congying Han, Hailin Shi, Tiande Guo, and Bowen Zhou. 2022. Petsgan: Re- thinking priors for single image generation. In AAAI.
Shikhar Sharma, Dendi Suhubdy, Vincent Michalski, Samira Ebrahimi Kahou, and Yoshua Bengio. 2018. Chatpainter: Improving text image generation us- ing dialogue. In ICLAW.
Yufan Zhou, Ruiyi Zhang, Jiuxiang Gu, Chris Tens- meyer, Tong Yu, Changyou Chen, Jinhui Xu, and Tong Sun. 2022. Tigan: Text-based interactive image generation and manipulation. In AAAI.
Jascha Sohl-Dickstein, Eric Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2015. Deep un- supervised learning using nonequilibrium thermody- namics. In JCML.
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2022. Multi-task pre-training for plug-and-ple task-oriented dialogue system. In ACL.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. 2021. Designing an encoder for stylegan image manipulation. TOG.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In NeurIPS.
Rui Wang, Peipei Li, Huaibo Huang, Chunshui Cao, Ran He, and Zhaofeng He. 2023. Learning-to-rank meets language: Boosting language-driven ordering alignment for ordinal classification. In NewrIPS.
Weizhi Wang, Zhirui Zhang, Junliang Guo, Yinpei Dai, Boxing Chen, and Weihua Luo. 2022. Task-oriented dialogue system as natural language generation. In SIGIR.

---

Appendix
Table 7: Variants of Dialogue Module.
Model
MLayers
#llcods
dk
de
deas
A CHATEDIT Benchmark Dataset
Small
20-1
512
Base
12
64
3072
708
Each sample in the CHATEDIT dataset contains an input image to be edited with the correspond- ing caption, the associated dialogue, and the user requests. Notably, similar to most image editing datasets, the dataset does not include ground truth manipulated images for each turn. Fig. 8 presents an example in CHATEDIT that contains a four-turn interaction. Fig. presents the web page for human paraphrasing, where the annotators are required to check and revise the dialogue to make the conver- sation fluent, natural, and consistent with the user requests.
Large
24
1096
1021
Table et al., 2020).
The breakdown of the StyleGAN2 (Karras
W+ layer index
Resolution
Layer name
# Channels
Conv
512
4x4
ToRGB
512
8x8
ConvO_ur up
512
8x8
Convl
512
8x8
ToRGB
512
16x16
ConvO_ur up
512
16
Conv1
512
16x 16
ToRGB
512
32x32
ConvO_un up
512
32x32
Convl
512
32x32
ToRGB
512
64x64
ConvO up
512
B Network Architecture
64x64
Convl
512
64
ToRGB
512
128x
ConvO_up
512
B.1 Architecture of the Dialogue Module
128x 128
Convl
256
11
128 128x122 128
ToRGB
256
The model used in the dialogue module is based on the pre-trained language model T5 (Raffel et al., 2020), which is a Transformer (Vaswani et al., 2017) cncoder-decoder framework. Each Trans- former layer comprises an attention mechanism and feed-forward network. Specifically, the at- tention mechanism is self-attention in the encoder layer and encoder-decodee attention in the decoder layer. The feed-forward network consists of dense layer with output dimensionality of du followed by a ReLU nonlinearity and another dense layer. For the base model, both the encoder and de- coder consist of 12 layers, where the "key" and "value" matrices of all attention mechanisms have inner dimensionality of d&v 64 and all atten- tion mechanisms have 12 heads. The output dimen- sionality of the first feed-forward network in each block dr = 3, 072 and the dimensionality of all other sub-layers and embeddings is dmodel = 768. The small model scales the base model by using 6 layers for the encoder and decoder. For each layer, it utilizes 8-headed attention, du = 2, 048, and dmodel 512. The large model has 24 layers for the encoder and decoder. It scales the base model up by using 16-headed attention, dr 1, 096, and dmodel 1, 024. Table summarizes the statistics of three models with different sizes.
256x256
ConvO_up
256
256 256
Conv1
128
13
256x256
ToRGB
128
512x512
ConvO_un _unvO_up
512x512
Conv1
64
512x512
ToRGB
64
ConvO up
1024 1024
Conv1
32
1024x 1024
ToRGa
32
section. Specifically, it generates images gradu- ally from low resolution to high resolution. Every major layer (every resolution) of the StyleGAN2 generator consists of two types of convolutional blocks: feature space convolutions (Conv), which are leveraged for feature map synthesis, and toRGB convolutions (ToRGB), which utilize convolutions to convert the feature map into RGB image. Each of these convolution blocks is modulated by vector of style parameters w. In our experiment, we utilized the W+ space where each W+ layer has its own style parameters w, E W. The details of StyleGAN2 (Karras et al., 2020) generator are listed in Table 8.
C Details of the Human Evaluation
Questions of the Human Evaluation. In the hu- man evaluations, we compare our framework with Talk-to over aspect of the manipulated image: manipulation, and three aspects on the gen- erated response of each turn in the dialogues: hu- manness, interestingness, and engagingness. The instructions for these four aspects provided to par- ticipants are shown as follows:
B.2 Architecture of the Image Editing Module
To supplement the description of the image edit- ing module, the architecture of StyleGAN2 (Karras et al., 2020) generator is described in detail this

---

User requirementa
User:
Pleaze
malnup: obe:
Systee:
Done Wheer about syong olick
Uter:
Good Please
maleag
Asir color bace hol:
Syaten:
u.S. abs
Use:
Make
makeup: skor
hair color 0439 fox: expression, ango,
Caption
System:
buaueo commmnn the broe hait
User:
Yop ouoo /east doe whar kollo Noo
makoig:: pole ski Nistor color Mack asprnssion w0g7
new andiig the searin fotkct
Figure 8: An example sample from the CHATEDIT dataset.
Formale
Plasee recia the nillowiig sloges
dralopaz of curmnt sum crre-ttic ornuiame nuptremenzs, adlect
dkilogse bask
Cannett Renieed dialogus
bore las
Spalem Ck, des De you vas ailiiaa wilme"?
Citt Vulk du harch coloows
Blacd
Collag bulky
hal?
Figure 9; Interface for the manual paraphrase.
Manipulation: Which one manipulates the im- age better with facial identity unchanged? Humanness: Which one sounds more natural and personable?
better in the setting that the single-turn method takes oracle user requests as input while ours take the dialogue as input and uses the dialogue module to obtain the tracked user requests.
Interestingness: Which one arouses your cu- riosity tells you something new or useful? Engagingness: Which one is more likely to capture your attention and make you want to further interact with it?
Visualization Results of Ablation study. In the ablation study of the main paper, demonstrate the effectiveness of the introduced dialogue mod- ule that extracts user requests to guide the image editing module quantitatively. We present quali- tative results in this section. We experiment with the multi-turn method that isn't equipped with a dialogue module to extract the user requests and thus directly takes dialogue as input. As shown in Fig 13, it fails to understand the user request accurately and ignores some of the user requests, suggesting the significance of our introduced dia- logue module in the multi-turn interactive editing.
We conduct the blind evaluation where participants will not be informed about the source of the manip- ulated images and generated responses (our frame- work or Talk-to-edit) to ensure fairness.
Inter-rater agreement. To evaluate the agree- ment among the answers of all participants, we calculate the inter-rater agreement score. The av- erage inter-rater agreement score is 0.26 in terms of Fleiss' kappa (Fleiss and Cohen, 1973), which demonstrates fair agreement.
Comparison Between Our Method with Talk- to-Edit. More qualitative comparisons between our method with Talk-to-Edit (Jiang et al., 2021) are presented in Fig. 14, 15, 16. Fig. 14 shows that there exists the attribute forgetting problem in Talk-to-Fad Editt. Fig. 15 illustrates that Talk-to-Edi Edit has an error accumulation problem. Besides, shown in these results, method can generate better responses with proper suggestions, which improves interactivity. Moreover, error judgment in the sule-based method will lead to the unexpected break-off of the interaction in Talk-to to-Edit, which is represented in Fig 16.
D More Qualitative Results
Comparison of Single-turr Editing and Our Pro- posed Multi-turn Editing. In Fig. 10, 11, 12, we present more visualization results to illustrate the attribute forgetting problem and error accumu- lation problem of previously single-turn methods, which can be avoided by our multi-turn approach. Specifically, utilize StyleCLIP as the image edit- ing method for both the sing-turn method and our multi-turn method for fair comparison. Notably, as shown In Fig. 11, 12, our method still performs

---

Single-tum (USR)
Single-tum (USR)
Multi-tur (USR)
Multi-turn
JUSay
Input
Gray hair
Bangs
Sad
Input
Lipstick
Blond hair
Smiling
(a) Attribute forgetting.
Aingled
Singge
Single-tum
JUSRy
Multi- Cit-tum (USR)
(UsR) Parziln
Input
Pale skin
Lipstick
Smiling
Input
Blond hair
Bushy eyebrows
Mustache
(b) Ervor sccumulation.
Figure 10: Comparison of single-turn editing and proposed multi-turn editing. Both methods take oracle user requests as input to make a fair comparison. (a) shows the attribute forgetting problem of the single-turn editing method. (b) illustrates the accumulation problem. By contrast, our proposed multi-tur editing approach avoids these issues.
Single-turn
(USR)
ours
System::
System
System give atfferest look mistache??
System: you want considuroble hoppy? User
inteise onger.
hoviop busty eyebrows aspoalling User it rephr lonk better this way.
rosy User: want
trylnp receding haitinc?
Uver Winl woat
esit that me toke
see what he would toot like Üser. It sqgudd be aicer this way, chhink
look, lind of would Mik to have adeebanns
with bland hov, could you help me with mor?
Singled (ogle-tuu -ttuum (USR)
Ours
Uier:: hove cook,
System: Ainihed Weslld
System: done. Would
Syrtem: Thir Shall? help you give new hatrost with bungs so your photo? Uneert might look better this way, mint.
System: it's done. Would you Wie change your mokeup?
System: Wil,, this Whhtt's neet?
Emmmi con you moke my
you Whe youself to more sou??
you Mele to experiment with buily cystrumu?? User: thhsk it might be nizer this way.
lis look gorgeous with
some Sprrict in the image?
User: goees would laok better this way.
User.Hopee /d vie what wosld took like bluck
Figure 11: Comparison of single-turn editing and proposed framework (a). Our method takes dialogue history as input and uses the dialogue module to track user requests while the oracle user requests are used for single-turn methods.

---

Single-turn (USR)
Ours
User: She
System: dispe incination
System: Bingo, hourtyler
Spstwmt it's done tut
System:: This
Lttte
fowords hosing harr User: Work in loow better (thinw
cheris unold lect ribe that beer User....
User: Make bangi not
ther: hent
now wnuld she wore Spunic.
Single-turn (USR)
Ours
User: twont to hove poir skio and thikk i books
Spadem, Ciey fiothed, is chare a prsaibilly of you apting for blnnd havr User: cot me take loat. Eroro miihht rice in that way.
System: Finithed. What slke de you wond udit? User: it enuld be biy smill.
Spatem Dave: Do you wore to see srltt ruyy chavis would lone tie your Uheniik, tharts, Faculd Vhe souee how would by We rhove gontes.
System: Firsttel it somcohing you'd Wike how Vnunik asprarence photo?
great.
Figure 12: Comparison of single-turn editing and our proposed framework (b), Our method takes dialogue history as input and uses the dialogue module to track user requests while the oracle user requests are used for single-turn methods.
Multi -t-uu
Muiti-tun
(Dial)
suvs
sur
User: wart nave bangs.
Systems: OK, do
System: Wiatt olse you wont manpukate?
System: OK What's noxt?
User: Male the smile bigger.
System: OK what else do want ny?
System: Fiusttod do System: Do nant to busly
want the porson to be fot neppor??
you mart pution Acunic
syelsros??
User: Nl, pleose change the hole oolor gray.
User: Aind of wont User. wear spucick.
User: I'd thk to soe whet' smoudd Nonk with frige
User: Al, wwart-2 see whre' seould book
User...
brmon
Multi-turn
Multi-turn (Dial)
SJnO
Ours
make you smiking the
edting
What's want edk?
about
ole
the enlor?
longer?
har??
bange longer?
Aie monipuoit??
User: Faar
User: No. the porson
User...
photo?
User: wond have Nostiol
User: Good laoa.
User....
beoi one.
dniko very haopll.
please give Xt by.
Figure 13: Visualization results of ablation study. The show the influence of the introduced dialogue module that extracts user requests from the dialogue on the manipulated images. The multi-turn approach doesn't equip with the dialogue module and thus takes the whole dialogue input of the image editing module.

---

User: Neee smilin User: User Ne te book lhen with hoad of how.
Cleer: User User Gar tuskng thene look anto you nop me with that? no amll his foce. halr the image?
sitee mustache
tim
Talk Talk-to-Editt
Talk Balk-to-Edh
Iiput
System:
System: maybe might
System: What's
Input
System: Fiillld.
System:
System: elgo you
good enough?
you
SJmO
SatO
Input
System: Okay this itll What else do you
System Done: What' about ryynn
System aingo What wèe do you madilplldd
Input
System: Okay done. De you mont eot! expreasion?
System: What else do you wont to moriduusts?
System: Fiolshed. unar about eatiig makoup?
big
Figure 14: Comparison of our method and Talk-to-Edit. These results demonstrate the attribute forgetting problem in Talk-to-Edi to--titt where the goatee attribute is lost the above cases.
User schood bigger atrlls.
User make
you moke my uppiear gray phodo?
User smiling hupply..
User mant bry mostleiii
brang lingae.
hoad
Talk-to-Edit
Talk- Mik-to-Ecl
Input
Systum: Are the fringo proper shape now?
System: This in What ceat?
System: is the senio of person just digt now?
Imput
System:
Sysnem: What alse maybe might would tie to edt?
System: What ave mayêe you nous lkk do manquett??
smile of the penson Just rigt now?
OHiS
Input
System: What else you want manipalite with?
System: Do you want to adti some wovers your hai??
System: It's donee Would you Ven expertmmnt with rody chaslee?
Inpot
System: Okay this is What wlse do you wont to kty?
System: Docn Do you want and some bongs in the iragee?
System it's done.
Woslt you New expertment with may chiwwaay
Figure 15: Comparison of method and Talk-to-Fahe These results demonstrate the error accumulation problem in Talk-to Edit, where artifacts occur and propagate to the final edited image.
User: lhe sno User Anti boegs to whit shoo would look hor.
User Sike d smiVing
User: Can you help dte add some baings to my hair
Üser: loand
User: The hair should be gray.
experiment with smikng expeession my image.
with gray noir.
phoio?
Talk-to-Edit
Talk:to-Eal Edtto--Edit
END
Input
System System: Finadied What whos moulf you lie ny?
Input
System: finge shope now?
System: Dse Whet sust?
System: anmla of the just right now?
Whet's sust?
ouno
Qur
Imput
System:: OK, what elso do you want chango?
System: Do you wart to catlig th expressior os well?
Systtm Whold you Uike me to help you actiove a warry nair?
Input
System: What else do you tont manpulate with?
System Do you wast to ads some waves to your nay?
System Wi's chone Would you Who experfmont with rody cheelts?
Figure 16: Comparison of our method and Talk-to-Edit. The left present case where the interaction is broken off unexpectedly Talk-to-Edd Edit due to its based method. END represents that the system terminates the interaction. The right present case where Talk-to- Edit fails to edit.

---

Table Illustration of out-of-distribution attributes.
Slot
Attribute
Expression
disgust, surprise, fear
Hair color
pink hair, purple hair, red hoir big eyes
Makeup
Table 10: Image editing performance on in-domain and out-of-domain test sets.
Test
FID
LPIPS
MinRel
AvgRel
domain
40.115
0.754
Out-of domain
40.264
0.458
0.739
0.759
E More Quantitative Results
E.1 Out-of-distribution Generalization
A distinct advantage of our proposed framework is its adaptability in utilizing various training strate- gies for the T5-based dialogue module. Drawing inspiration from (Lin et al., 2021), cast the slot value tracking question-answering task, which promotes enhanced generalization on unseen data, For example, the model is prompted with queries like "what is the hair color?"
For evaluation, we use an out-of-distribution (OOD) test set comprising 1,000 samples based previously unseen values, as detailed in Table 9. Experimental results suggest model's robust- ness against OOD scenarios during testing. The dialogue module achieves accuracy of 58.90% on the OOD test set. As for image editing, the re- sults on the OOD test set align closely with those the in-domain test set.
E.2 Ablation Study of Image Editing Module.
We performed ablation studies for both identity loss (Iaa) and L22 loss within the image editing module. The results are presented in Table 11. is observed that both loss types influence the image-editing per- formance, with T.2 loss having more significant impact. This likely because L2 loss constrains the degree of feature vector transformations hid- den space, and significant changes in StyleGAN's hidden space can introduce severe artifacts.
Table 11: Ablation study of image editing module.
Image Editing
Method
Input
FTD
I.PIPS
MinRel
AvgRel
Multi-tum
USR
40.115
0.449
0.754
0.773
w/o Zaa
USR
40.475
0.452
0.742
0.763
w/o lgg
USR
74.639
0,794
0.715
0.752


Reviews:
[{"review_id": "ekit4ppyPe", "reviewer": "Reviewer_893r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your insightful feedback on our paper. We would like to clarify that our work distinguishes from Talk-to-Edit in several significant ways:\n\n1. *New Benchmark Dataset for Multi-Turn Interactive Image Editing.* The Talk-to-Edit paper focuses on single-turn interactions, largely hindered by the absence of a suitable multi-turn benchmark dataset. We address this limitation by introducing a novel dataset tailored for multi-turn interactive image editing. The dataset includes multi-turn conversation and annotations for image editing. We also provide a set of evaluation criteria for assessing multi-turn interactive image editing performance.\n\n2. *Enhanced User Engagement and Robustness.* Talk-to-Edit relies on a **rule-based approach to returning templated response for single-turn interactions** and often encounters bugs that lead to abrupt exits. We tested this by randomly selecting 100 dialogues, finding that Talk-to-Edit broke off in 73% of cases (see Fig. 17 in the main paper). In contrast, our framework utilizes a language model-based dialogue module, allowing for more coherent and diverse user interactions. It is also able to more reliably extract user edit requests from the whole conversation instead of from each single turn utterance.\n\n3. *Mitigation of Error Accumulation and Attribute Forgetting Issues.* Talk-to-Edit treats multi-turn editing as a sequence of successive single-turn edits, which leads to issues like error accumulation and attribute forgetting. Our framework alleviates these issues by closely integrating a dialogue module that extracts current user edit requests from the full multi-turn conversation. These extracted requests then guide the image editing module. We supplement a quantitative comparison between our method and Talk-to-Edit on 100 randomly-selected cases. Experimental results are shown in Table r1. Our framework achieves better image quality with lower FID and LPIPS scores, highlighting reduced error accumulation. Moreover, our method excels in MinRel and AvgRel metrics, underscoring its better alignment with user requests and effective mitigation of attribute forgetting.\n\n**Table r1: Quantitative comparison of the image editing performance between ours and Talk-to-Edit.**\nModel | FID ↓ | LPIPS ↓ | MinRel ↑ | AvgRel ↑\n-------- | -------- | -------- | -------- | -------- \nTalk-to-Edit | 132.555 |0.644 |0.731 |0.739\nOurs |98.760 |0.439 |0.770 |0.776", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and easy to follow. The interactive multi-turn image editing dataset is a good contribution to the research community. The authors provide limited ablation studies in the main body of the text.", "reasons_to_reject": "The work seems to be an incremental and marginal improvement on prior work( e.g talk-to-edit). The authors show some qualitative comparison with talk-to-edit in the Appendix, which is somewhat convincing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors present a dataset that is derived from Celeb-QA that supports interactive facial image editing via dialogue. The authors incorporate annotated multi-turn dialog. The authors' dialog state tracking solutions take the overall conversation up to the image editing request turn into account, instead of only looking at a single previous turn. The authors provide ablation studies to show the effectiveness of their approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6OZwxcwEs0", "reviewer": "Reviewer_zAak", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Please find our response below:\n> **Q1** for the interactive user intention understanding, ChaGPT could fully handle this task with well-designed prompt, seems the adopt T5 is unneeded. \n\nThe reasons of the choice of T5 over ChatGPT for User Request Tracking are two-folds:\n- **Performance**: We've conducted a quantitative comparison between our T5-based dialogue model and ChatGPT on 100 randomly selected dialogues. Our method achieves a Joint Accuracy of 88.86%, outperforming ChatGPT's 76.26%. Specifically, we used the prompt settings proposed by Heck et al. [r1], which is up-to-date and has achieved impressive results on the MultiWOZ dataset.\n- **Practical Considerations**: The performance of ChatGPT often hinges on intricate prompt engineering and comes with significant API costs. Additionally, privacy and policy constraints may limit its applicability. Therefore, our model, designed for local deployment, serves as a more practical alternative.\n\n> **Q2** no novelity is notified in the dialogue/image generation system construction.\n\nWe introduce a new benchmark dataset for multi-turn interactive image editing along with a baseline for handling this task. \n\nOur approach innovatively integrates a task-oriented dialogue module with an image editing module. Compared with prior work that reduces multi-turn interactions to successive single-turn interactions, our model supports multi-turn engagement and editing. The dialogue module extracts the user edit requests and generates coherent and fluent responses given the full conversation context as input. The extracted user edit requests are derived from the whole conversation instead of turn-level updates, which effectively mitigates issues of **error accumulation** and **attribute forgetting** common in single-turn approaches. \n\nOur experimental results, both quantitative and qualitative, showcase the advantages of our multi-turn framework. The results suggest that our proposed approach serves as an effective baseline for multi-turn interactive image editing tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed system has involved a list of components for user interactive intention understanding, including an adpot T5 dialogue system and ChatGPT, providing reasonble overall quality. ", "reasons_to_reject": "1, for the interactive user intention understanding, ChaGPT could fully handle this task with well-designed prompt, seems the adopt T5 is unneeded.\n2, no novelity is notified in the dialogue/image generation system construction.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces an interactive facial image editing tool. This system is constructed with a task-oriented module and an image editing module, and can modify given image following the user requirements in the dialogues.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KVjVwgbqMX", "reviewer": "Reviewer_Ht4U", "summary": "", "questions": "- L244: what is “predefined policy”? I think this detail is important and should be described sufficiently", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments.\n> **Q1** *If the current approach is still robust with unseen slots/ slot values during test time.*\n\nAn advantage of our proposed approach is its flexibility in adopting different training strategies to train our T5-based dialogue module. Following [r2], we formulate the slot value tracking as question answering task to enable better generalization performance on unseen data. Specifically, we will ask the model for the value for each slot, such as \"what is the hair color?''. Our tests reveal relative robustness against out-of-distribution (OOD) scenes during test time. \n\nSpecifically, we evaluate the out-of-domain performance across various scenarios as detailed in Table r2. We construct a new OOD test set containing 1000 samples based on these unseen values shown in Table r2. For the dialogue module, without modifying our model, 58.90% accuracy is achieved on the OOD test set. For image editing, as shown in Table r3, the performance on the OOD test set is comparable to that of the in-domain test set.  There is still room for improvement in performance on the OOD test set. As discussed in the Limitations section of the main paper, our future research will explore to make an improvement in this topic.\n\n**Table r2: Illustration of added out-of-distribution attributes.**\nSlot     | Attribute\n-------- | --------\nExpression  | disgust, surprise, fear\nHair color  | pink hair, purple hair, red hair\nMakeup   | big eyes\n\n**Table r3: Quantitative comparison of the image editing performance between in-domain and out-of-domain test sets.**\nTest set | FID↓ | LPIPS↓ | MinRel↑ | AvgRel↑\n-------- | -------- | -------- | -------- | -------- \nIn-domain | 40.115 | 0.449 | 0.754 | 0.773\nOut-of-domain | 40.264 | 0.458 | 0.739 | 0.759\n\nReference\n\n[r2] Lin, Zhaojiang, et al. 2021. Zero-Shot Dialogue State Tracking via Cross-Task Transfer. In EMNLP.\n\n> **Q2** *Additional study could be provided.*\n\nWe performed ablation studies for both identity loss (Lid) and L2 loss within the image editing module. The results are presented in Table r4. It is observed that both loss types influence the image-editing performance, with L2 loss having a more significant impact. This is likely because L2 loss constrains the degree of feature vector transformations in hidden space, and significant changes in StyleGAN's hidden space can introduce severe artifacts.\n\n **Table r4: Ablation study.**\nMethod | Input | FID↓ | LPIPS↓ | MinRel↑ | AvgRel↑ \n-------- | -------- | -------- | -------- | -------- | -------- \nMulti-turn | USR | 40.115 | 0.449 | 0.754 | 0.773 |\nw/o Lid | USR | 40.475 | 0.452 | 0.742 | 0.763\nw/o L2 | USR | 74.639 | 0.794 | 0.715 | 0.752\n\n\n> **Q3** *What is “predefined policy”?*\n\nThe term \"predefined policy\" refers to a set of rules that govern the dialogue flow to ensure it aligns logically with the user's instructions and system functions. For instance, if a user requests a change in hair color to blond, the system's predefined policy would prevent it from generating a redundant or illogical suggestion like \"Do you want to dye your hair blond?\". This policy serves to prevent unrealistic or redundant conversation scenarios. We appreciate the call for clarity and will refine this section in our paper accordingly.\n\n> **Q4** *Missing References.*\n\nThanks for your valuable suggestion. We will incorporate the suggested citations into the relevant sections of our paper in the updated version.\n\n> **Q5** *Typos grammar style and presentation improvements.*\n\nThank you for pointing out the difficulty in interpreting Figure 2. In the single-turn approach, the cascaded errors in the single-turn mainly involve unintended changes to the gender and eye makeup. We will aim to improve the clarity of these illustrations in the updated version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Large-scale benchmark of 12k images, using the annotation from the Celeb-HQ dataset. Comprehensive analysis of the dataset, including a manual review to ensure the dialogue flow logic. \n- The proposed approach is more robust by tracking the user requests from the entire dialogue history in the form of state tracking. This can help to avoid error cascading and accumulation over multiple turns of interactions ", "reasons_to_reject": "- The editable attributes are still quite limited to only 4 types of slots (expression, hair color, hair, and makeup). There are many variances in the real-world and many slots would be missing in the annotations. I wonder if the current approach is still robust with unseen slots/ slot values during test time. \n- Additional study could be provided e.g. to test the model in a zero-shot setting with unseen attributes. More ablation could be done (e.g. by model hyperparameters, or losses) to make the results more convincing. ", "questions_to_authors": "- L244: what is “predefined policy”? I think this detail is important and should be described sufficiently", "paper_topic_and_main_contributions": "The authors propose a new task and benchmark for interactive facial image editing through dialogues. A new benchmark called ChatEdit is created to evaluate AI models in image editing and conversation response generation. The benchmark contains useful annotation to track user requests and generate responses. \n\nA new model is also proposed to perform this task end-to-end, including a dialogue module and an image editing module. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "UniConv: A Unified Conversational Neural Architecture for Multi-domain Task-oriented Dialogues, EMNLP 2020 \n\nNon-Autoregressive Dialog State Tracking, ICLR 2020 "}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In Figure 2, It is quite difficult to see the difference in generated images between single-turn and multi-turn. What type of error is cascaded in the single-turn and not in the multi-turn approach? "}}]

Author rebuttal:

Meta-review: The work provides a dataset and framework for interactive facial image editing through dialogue.

**Pros**: Reviewers agree the dataset will be a good resource to the community. The majority of reviewers also agree the dialog state tracking solution is also a new (if incremental) contribution that is well defended empirically.

**Cons**: Some reviewers raise concerns that the work is incremental compared to previous research (e.g., talk-to-edit and the dialogue/image generation components).
Decision: Accept-Main