Localize, Understand, Collaborate: Semantic-Aware Dragging via Intention Reasoner
Authors: Xing Cui
Pei Pei Li
Zekun Li
Xuannan Liu
Yueying Zou
Zhaofeng He
Venue: NeurIPS 2024 poster (2024)
Keywords: Image editing
Diffusion model
Large Language Model
Primary area: diffusion_based_models
TL;DR: We introduce a methodology for semantic-aware image dragging with high image fidelity.

Abstract: Flexible and accurate drag-based editing is a challenging task that has recently garnered significant attention. Current methods typically model this problem as automatically learning "how to drag" through point dragging and often produce one deterministic estimation, which presents two key limitations: 1) Overlooking the inherently ill-posed nature of drag-based editing, where multiple results may correspond to a given input, as illustrated in Fig.1; 2) Ignoring the constraint of image quality, which may lead to unexpected distortion.
To alleviate this, we propose LucidDrag, which shifts the focus from "how to drag" to "what-then-how" paradigm.  LucidDrag comprises an intention reasoner and a collaborative guidance sampling mechanism. The former infers several optimal editing strategies, identifying what content and what semantic direction to be edited. Based on the former, the latter addresses "how to drag" by collaboratively integrating existing editing guidance with the newly proposed semantic guidance and quality guidance.
Specifically, semantic guidance is derived by establishing a semantic editing direction based on reasoned intentions, while quality guidance is achieved through classifier guidance using an image fidelity discriminator.
Both qualitative and quantitative comparisons demonstrate the superiority of LucidDrag over previous methods.

Localize, Understand, Collaborate: Semantic-Aware Dragging via Intention Reasoner
Xing Cui', Peipei Li Zekun Li?, Xuannan Liu', Yueying Zou', Zhaofeng He'
Beijing University of Posts and Telecommunications
Univerrityy California, Santa Barbara
(cuixing, lipeipei, liuxuannan, zouyueying2001, zhaofenghe) @bupt.edu.cn zekunli @cs.ucsb.edu
"wo horses. Target prompt: "Two horses with necks."
prompt:
"Two horses."
prompt:
"Two florses for away.
prompt:
"Two horses with heads
Target prompt:
Target prompt: "Two horses close
Image User Edit
Diverse results generated by our LucidDrag.
Image & User Edit
LucidDrag
Image & User Edit
LucidDrag
Image & User Edit
LocidDrag
Given an input image, the user draws mask specifying the editable region and clicks dragging points (handle points (red) and target points (blue)). Our LucidDrag considers the ill-posed nature of drag-based editing and can produce diverse results (the first row). Besides, it achieves outstanding performance in editing accuracy and image fidelity (the second row).
Figure I:
Abstract
Flexible and accurate drag-based editing is a challenging task that has recently garnered significant attention. Current methods typically model this problem automatically learning "how to drag" through point dragging and often produce one deterministic estimation, which presents two key limitations: 1) Overlooking the inherently ill-posed nature of drag-based editing, where multiple results may corre- spond given input, illustrated Fig. T 2) Ignoring the constraint of image quality, which may lead to unexpected distortion. To alleviate this, we propose Lu- cidDrag, which shifts the focus from "how drag" "what-then-how" paradigm. LucidDrag comprises an intention reasoner and a collaborative guidance sampling mechanism. The former infers several optimal editing strategies, identifying what content and what semantic direction to be edited. Based on the former, the latter addresses "how to drag" by collaboratively integrating existing editing guidance with the newly proposed semantic guidance and quality guidance. Specifically, semantic guidance derived by establishing a semantic editing direction based on reasoned intentions, while quality guidance is achieved through classifier guidance using an image fidelity discriminator. Both qualitative and quantitative comparisons demonstrate the superiority of LucidDrag over previous methods. Code is available at: :attps://githus com/ wttccoocccciieeeeee /nixxingg1087////cciiirgggg Dou.
*Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The impressive success of diffusion models 27 12
has promoted the advancements in the field of image editing [26 69 I7. The conventional paradigms for editing conditions typically encompass
text [1] 47 25}, instruction 7 68 [6], or image [64 61 21.. However, these conditions prove inadequate in effectively communicating specific image aspects, such as shape and location [IS].
To address this, recent studies [14
50 53 define a new task called drag-based editing, which incorporates dragging points as conditions. These studies specifically regard drag- drag-based editing as the problem of "how to drag" and tackle it by designing an editing guidance loss, enabling the model to implicitly learn the appropriate solutions. Despite their considerable success (50] 45, 46), these methods have two limitations: Firstly, they neglect the inherent ambiguity of semantic intention. baag-baeed editing is an poo-posed problem. As depicted in Fig. [ the drag points starting from the horse's 's head and ending at its upper right can indicate various semantic intentions, such as "make the neck longer," "raise the head," or "bring the two horses closer." However, existing methods mainly follow point dragging principles [53] 45), focusing on positional movement by constraining feature correlation between the source and target points. This current position optimization strategy inherently overlooks semantic diversity, making it challenging to generate images with precise semantic perception. Secondly, they overlook the preservation of the overall image quality. Current methods prioritize editing accuracy while neglects the overall image quality. Some methods [45] 46) utilize score-based classifier guidance for image editing, which can cause mismatches between the distribution of the edited image and the input image, compromising image fidelity.
In this study, we divide the drag- brag-based editing task into two steps. We introduce preliminary step, "what to drag", to determine the specific content and semantic direction for editing before addressing "how to drag". That is, we shift the focus from "how to drag" to a paradigm of As shown in the first row of images in Fig. T before editing, we need to determine what we are going to edit, such as the horse's 's head, and the semantic strategy for editing it towards the upper right. For example, we could make the horse lift its head, elongate its neck, or shorten the distance between the two horses. With the "what to drag" information established, we can then proceed to address "how to drag". " To achieve this, we construct an intention reasoner that integrates a Large Language-Visin Model (LVLM) and a Large Language Model (LLM) to deduce possible intentions. As illustrated in Fig. D given an input image and drag points that begin at the horse's face and end in its upper-right, the intention reasoner acts
AI agent to infer potential intentions, subsequently providing corresponding source and target prompts. Once the intention is determined, we then address "how to drag" by injecting the reasoned semantic information into the model by developing collaborative guidance sampling, which integrates editing guidance with the proposed semantic guidance and quality guidance. Specifically, the semantic guidance is derived from the source and target prompts determined by the intention reasoner. The asymmetric prompts establish a clear semantic editing direction towards the target intention. Additionally, discriminator employed as the score function to provide quality guidance. The quality gradient is generated based on image fidelity and incorporated into the model via the classifier guidance mechanism. For the editing guidance, we follow previous work 45 to maximize the feature correspondence between the source and target positions.
LucidDrag is an intuitive framework for wwhat-then-how drag-based editing, demonstrating outstand- ing performance in terms of semantic perception ability, diversity, and editing quality. LucidDrag enjoys several attractive attributes: Firstly, clear, diverse, and reasonable semantic intentions. Our innovative method employs LVLM and LLM to explicitly deduce the intention by localizing the drag points and reasoning several probable intentions. By explicitly incorporating semantics, we enhance semantic perception and offer diverse editing modes, enriching the variety of outcomes. Secondly, enhanced overall generation quality. By introducing collaborative guidance sampling, we significantly promote the generation quality of trag-based editing. Specifically, we achieve diverse and accurate image editing by introducing an additional semantic editing direction through semantic guidance. Additionally, we maintain better image quality by explicitly constraining the image distribution using quality guidance. In summary, our main contributions are:
We propose a new "what-then-how" paradigm for drag- based editing and introduce Lucid- Drag. To address the "what" problem, we present an intention reasoner, which employs LVLM and LLM to determine what content and semantic direction should be edited.

---

We then use the inferred semantic results of "what" to guide "how", enhancing editing accuracy and overall image quality. Additionally, quality discriminator also employed to provide quality gradients via score-based classifier guidance. This quality guidance. combined with editing and semantic guidance from "what," improves the precision and fidelity of the results.
We present quantitative and qualitative results demonstrating the applicability and superiority of our method, in terms of editing aceuracy, fidelity, and diversity.
2 Related Work
2.1 Diffusion Models
20 56 aims to estimate the noise added to the image 2 0z + Ozc: where ar and og are non-learned parameters. The training loss is to minimize the distance between the added noise and the estimated noise:
Diffusion model 54
((0) =
c (2)
where / refers to the time step, is the ground-truth noise, y is an additional condition. Diffusion models can be regarded as score-based models (56). In this context, co serves as an approximation of the score function for the noisy marginal distributions: cœ(2)) = V., p(z).
can sample images given conditioning y by starting from random noise 2y ~N I). and then alternating between estimating the noise component and sampling 2 The noise estimated as:
co(21 t, 8). é
(2)
The sampling could be based on DDPM [00 or DDIM (55]. this paper, we utilize DDIM, which denoising ž previous step ž with deterministic process:
De-i
21-1
a
2.2 Classifier Guidance
Classifier guidance moves the sampling process towards images that are more likely according to the classifier [12]. As a powerful conditional sampling strategy, has been used various tasks,
including generating diverse results [8) refining generative process 35655 and image editing [45] 46).
Specifically, combines the unconditional score function for p(z:) with a classifier p(y|zc) to produce
samples from p(=llv) p(v]=1)p(z)) [12 [66] Classifier Guidance requires access to a labeled dataset and the training of noise-dependent classifier p(y|2)) which can differentiated concerning the noisy image 24. During the sampling process, classifier guidance can be incorporated follows:
Vx, log (F=lv) Vz, log ((x) + V., log ((v)==),
(4)
The first term is the original diffusion denoiser, and the second term refers to the conditional gradient produced by an energy function g (z:; t, y) g (y]+z). Thereby, we apply classifier guidance by modifying εg:
é ex L v) + 1 V logg (zrz L (,
(5)
where 7) is an additional parameter parameter that modulates the strength of the guidance.
2.3
Visual Programming
demonstrated remarkable emergency abilities || explore to leverage them for planning and reasoning in multi-modal image generation. For example, ChatEdit [
As LLMs
49
58 and LVLMs
74
40
researches [18
37 59
utilizes pre-trained language models to track the user intents. Some approaches [88 37, 59 augment the original prompt through paraphrasing. Recent,
visual programmer methods [22 24 translate complex input prompts into programmatic operations and data flows. Despite the effectiveness of these strategies in augmenting the input text instructions, they overlook the capacity to reason within visual-modal instructions, such as the dragging points in drag editing tasks. In contrast, our LucidDrag addresses this gap by integrating LVLM and LLM.
3

---

What to drag
How to drog
prompt: wamon looking
Target prompt:
LVLM-driven locator
"The of woman.
LLM-driven understander
Intention:
Source prompt "A woman. " Tanges prompe. "A wanrer looking sideways."
"Wake the looking sideways.
Intention:
"Wove the the left."
Intention Reasoner
prompt: womon." TTrggt prompt: "A woman
left."
dd
"A woman looking up.
Semantic Guidance
"A woman looking down." "
Encoder
Decoder
DDIM Inversion
U-Net
Self- Artention
Meta Info "A woman.
Editing
Quality
Quality
captron:
Guidance
discriminator
Guidance
points
5dit
PuQuoliyty
mask:
Collaborative Guidance
Figure 2: Overview of Lra.dDrag. LucidDrag comprises two main components: intention reasoner and a collaborative guidance sampling mechanism. Intention Reasoner leverages an LVLM and an LLM to reason N possible semantic intentions. Collaborative Guidance Sampling facilitates semantic- aware editing by collaborating editing guidance with semantic guidance and quality guidance.
2.4
Image Editing
Image editing aims to manipulate an image according to specific conditions. Prior methodologies [3
36
can only manipulate specific attributes. The prevailing approaches have primarily relied text conditions. For example, [9] 2] manipulate images in the GAN [29] latent space by learning an
3
edit direction. Motivated by the success of the diffusion model (7 60). state-of-ther -fetttr-rt methods extend their exploration into diffusion-basee
image editing by exploring the initial noise [O, 43
attention maps 26 63 15), or prompts
57 Recently, DragGAN 50 explores
novel editing scheme that drags any points of the image to reach target points with the help of StyleGAN [29] latent space. FreeDrag (38 improves point tracking by introducing adaptive feature updating and backtracking, Readout Guidance [42] solves the challenging task by leveraging the video dataset. The following works [45] 46) utilize the feature correspondence to direct the editing process. Nevertheless, they only use the drag points control, which is insufficient due to the potential diversity semantic intentions. to achieve semantic-awe aware editing. The intention reasoner reduce cognitive load, handle vague requests, and discover potential needs. Additionally, generates precise descriptions automatically, ensuring accurate and consistent manipulations.
contrast, our approach introduces an intention reasoner
Method
In this section, we introduce LucidDrag, unified framework for Drag Manipulation via Localing, Understanding, Collaborate Guiding. Within our framework, image drag editing is decomposed into two stages. Firstly, the Intention Reasoner translates the user-drag points into potential semantic intentions and generates corresponding source prompt and target prompt, thus solving the problem of "what edit". Then, the Collaborative Guidance Sampling designed facilitate image editing. Concretely, prompts generated from the intention reasoner are utilized in the DDIM Inversion and Diffusion U-Net, providing semantic guidance for the generation of semantically controllable results. Besides, LucidDrag employs discriminator to provide quality guidance for improved image fidelity. The semantic guidance and the quality guidance collaborate with the original editing guidance, offering a novel perspective of "how to edit". We elaborate on more details of our techniques below.
3.1 Intention Reasoner
As depicted in Fig. [ the first stage of our LucidDrag is the intention reasoner, which bridges the gap between the input point condition and semantic intention. This intention reasoner consists of two key

---

components, i.e., an LVLM locator that identifies the interested position, and an LLM understander that interprets input conditions into semantic intentions.
LVLM-driven loactor Given an input image, it may contain various objects situated different positions. To accurately identify the objects of interest, we employ an off-the-shelf pre-trained large vision-language model (LVLM) Osprey [67] Osprey is trained with fine-grained mask regions, enabling it to comprehend images at the pixel level. With the input imagez, we instruct the Osprey model with drag points P generate descriptive representation of the objects of interest O, i.e,, O = LV LM (z, P). As shown in Fig. 2 O="The nose of a woman", which subsequently serves as input to prompt the LLM to understand the condition and reason potential intentions.
LLM-driven reasoner As shown in Fig. each point condition may encompass various semantic intentions. For instance, it may represent non-rigid manipulation. such as "looking down", while may also represent rigid manipulation, such "moving to the left". Our LLM-driven reasoner is designed to discern potential semantic intentions to facilitate semantic-aware drag-base based editing. We leverage the capabilities of the large language model, GPT 3.5 [49], acting as an AI agent to reason the possible intentions. We take the generated description of the object of interest O, the original image caption C, and drag points P as input. Then, we prompt the LLM with in-context examples to generate N possible intentions, i.e. D = (LM C, P), where each output sample can have different intentions or levels of complexity. Specifically, D {[dy, P (d))) where d, = {i,.P% Pj. p3 } is the generated text output. iy. pj. p` represents the predicted intention, the predicted description of the source image (source prompt), and the predicted description of the target image (target prompt), respectively. P (dj) is the corresponding confidence probabilities of the j-th generate text output. Finally, we select outputs {d.- ""=))== by sampling based the confidence probabilities. The confidence probability reflects the quality of the output. A higher confidence probability indicates that the intention is more reasonable, leading better editing results.
(o= argmax (P (d:).
(6)
d,ED
3.2 Collaborative Guidance Sampling
As shown in Fig. 2 the objective of collaborative guidance sampling is to modify the intended content while ensuring the preservation of irrelevant components. The input image is inverted to noise 20th through DDIM Inversion [55] 44 During the inversion process, the intermediate noise .qud 2t along with the corresponding key h:!ud and value of"d of the self-attention layer, recorded in the memory bank, which serves in guiding subsequent generation process. Subsequently, we generate the edited images employing collaborative guidance sampling which incorporates three fundamental components: semantic guidance, quality guidance, and editing guidance. Each of these components contributes to the overall editing process distinctively, thereby ensuring a balanced and comprehensive approach to image editing.
Semantic guidance As textual conditions can convey semantic information, we leverage the source and target prompts generated by the intention reasoner to facilitate semantic-aware dragging. Specifically, during the inversion process, we employ the source prompt to transform the input image into its corresponding noise by iterating DDIM inversion, i.e., 2 = DDIM inversion (2; During the sampling process, we utilize the target prompt to generate the target image, i.e., =gen = DDIM (=îen ...)).))).. .. As there exists a divergence between the source and target prompts, the asymmetric textual condition introduces distinct editing direction that is oriented toward the target image, thereby offering semantic guidance. This differentiation facilitates a semantically guided editing process, enhancing the semantic coherence of the edited image.
(z. p.))
Quality guidance As shown in Fig. 2 we design quality guidance to ensure the quality of the generated image. discriminator is trained to distinguish between high-fidelity images and low- fidelity images at any step l. In particular, given real image and its corresponding text description y. we utilize a stochastic process to simulate potential points of drag and their respective directions. Then, we generate images using DragonDiffussin [45.. The images with an aesthetic score [52)
5

---

below 5, are classified as wwwfiideiity. Their corresponding real images are considered The selected low-fidelity images. along with the figh-iidelity images. constitute the training dataset of the discriminator, Finally, the training dataset comprises total of 10,000 high-fidelity images and 10,000 low-fidelity images.
As the intermediate representation of the diffusion U-Net captures semantic information of the input image [32]. we utilize this hidden representation evaluate image quality. The discriminator com- prises the down block and middle block of the Diffusion U-Net to capture the semantic information, followed by linear classifier layer. During training, we froze the down blocks which are initialized with Stable Diffusion v2.1 1-base [ST]. The middle block and the prediction layer are fine- fine-tuned to classify images as real or fake. The conditional discriminator d( d(x is trained by minimizing the canonical discrimination loss:
L = By. (Raap(t (n) log d(= Q)]
(7)
(0 log(1 d( y: 0)]]
The energy function to constrain the image quality is defined as
d' (= ww;
Iquality ((<11))
d' ()
(8)
Editing guidance Following DragonDiffusion 445 we extract intermediate features Fr and Fquud from 2 and zqui via UNet denoiser Ea respectively. The energy function is built by calculating the correspondence between Eg and Findd The editing guidance includes editing target contents and preserving unrelated regions. We denote the original content position, target content position, and the unrelated content position as binary masks, i.e., orig m far and m share, respectively, The energy function to drag the content is defined as:
gdrag a + 3 SSFFF matar, Exqud morig)'
where S(Fyen ma Frud m mor)) calculates the similarity between the two regions of Fren and Forud Similarly, the energy function preserve the unrelated region is defined as:
Geontent
(10)
+ 3 Stooet Fgge sharee Eggu mohere)
The final editing energy function defined as:
Gedst We Idrag We Icontent:
(11)
where we and we are hyper-parameters to balance these guidance terms. Finally, the editing guidance collaborates with the quality guidance during sampling:
g (#x; t, y) 9edst + Iguality.
(12)
Additionally, following [.] to ensure content consistency between the edited image and the input images, we replace the keys and values within the self- oeff-attention module of the UNet decoder with those retrieved from the memory bank.
4 Experiments
4.1
Implementation Details
To train the quality diseriminator, we employ the Adam optimizer with a learning rate le-4. We set the training epochs as 100 and the batch size as 128. For the denoising process, we adopt Stable Diffusion [11] as the base model. During sampling, the number of denoising steps is set
= 50 with a classifier-free guidance of 5. The energy weights for Squality- Idrag and yeontent are set to Te 3, 4 and 4e 4, respectively. The training of the discriminator can be conducted on
NVIDIA V100 GPU and the inference can be conducted NVIDIA GeForce RTX 3090 GPU.

---

Image
Source prompt: "A
Source prompt: face,"
Source
looking forward. Targot prompt: "A cute cat looking down."
Target prompt: cuto car in the bottom of the image. "
Target prompt: "A cute cat with long face."
Image
Source prompt: "A bud.""
Source prompt: bud short branch,"
Target prompt: blooming flower."
Target prompt: "A big bud.
Target prompt: "A bud on long branch,"
Image Edit
Source prompt: "A short mushroom."
Source prompt: "A mushroom. prompt: jot mushroom.
Source prompt: "A mustroom In the bottom." * Target prompt: mushroom In the middle.
Target prompt: "A tall mushroom.
Figure 3: LucidDrag allows generating diverse results conforming to the intention.
countryside
scene with house and high target: countryside scene with house and low gross.
of
bird with its wing middle position.
target: photo of bird with its wing raised.
source: photo of with big hole.
torget: photo leaf with a small
source: a wide building with blue sky background. thin building with blue background.
Image&Edit DragDiffusion
FreeDrag
DragonDiffusion
DiffEditor
Ours
Qualitative comparison between our LucidDrag and other methods in drag-based editing.
4.2 Comparisons
Semantic-aware dragging Since our LucidDrag effectively discerns potential intentions, first evaluate its semantic-awarent ability. Specifically, given an input image and corresponding dragging conditions, sample several intentions and obtain corresponding results. The results are shown in Fig. B On the one hand, the intention reasoner deduces reasonable intentions that align with the input dragging points and generate semantic-aware images, demonstrating both enhanced understanding of semantic intentions and increased diversity. On the other hand, our method can generate high-fidelity images aligned with the input prompts, improving the quality of the results.
Content dragging
We evaluate the proposed LucidDrag against existing drag editing models 38
53 45
We first conduct quantitative comparisons. Following DragDiffusion (53, we utilize the DragBench benchmark which designed for the image-dragging task. In DragBench, each image accompanied by set of dragging instructions, including several pairs of handle and target points
46

---

Table 1: Comparisons of content dragging DragBench.
DrugDiffusion FreeDrug DragonDiffusion
DiffEditor
Ours
Mean Distance (I)
32.60
30.40
29.16
26.01 +: 0.81 20.46 1 0.77
GScore (1)
6.94
7.15
6.26
7.37
and mask indicating the editable region. For editing precision, we use the Mean Distance [50 to evaluate the model's ability to move the contents the target points. For image quality, existing Image Quality Assessment methods (30, (70) rely on handcrafted features or are trained on limited image samples, which do not always align well with human perception [72]. Thereby, we employ GScore 72 to provide human-alin human-aligned assessment of image quality via Large Multimodal Models.
The quantitative results presented in Tab. [ For dragging precision, LucidDrag consistently outperforms other methods by significant margin across all categories in Mean Distance, indicating higher accuracy in dragging handle contents target positions. In terms of image quality, our method achieves an average GScore of 7.37, surpassing DragDiffusion (6.94), FreeDrag (7.15),
DragonDiffusion (6.26), and DiffEdito (6.42).
We also present qualitative results in Fig. 4.2 DragDiffusion and FreeDrag has difficulty in accurately dragging corresponding contents designated target locations. Although DragonDiffusion and DiffEditor can better recognize handle points and achieve more precise point movement, they tend to introduce artifacts, thereby reducing image quality. In contrast, LucidDrag demonstrates superior drag control by understanding potential intentions and providing semantic guidance. Furthermore, LucidDrag generates images with greater fidelity due to the explicit quality guidance it provides.
Object moving We also conduct experiments on the object moving task, which can be seen special task of drag-style manipulation [45]. We compare our method with DragonDiffusion [45] and DiffEditor [46). Following DragonDiffusion [45]. select 20 editing samples the test set. We calculate the CLIP distance between the edited results and the target description. Besides, inspired by (72), utilize Large Multimodal Models evaluate the overall performance, denoted as the LMM score. The results are shown Tab. Our approach achieves higher CLIP scores and LMM scores, demonstrating the promising performance of our method.
Source prompt: A photo of many cups on the table, with cup on right.
Target prompt:
A photo of many on the table,
cup an the left.
Source prompt: An astronaut plays
football, footbal!
on his left.
Target prompt: plays football, The footbal! is his right.
Source prompt: photo of four apples. Target prompt: photo apples on apple top right.
prompt: donut on square board.
Target prompt: donut an the left of square board.
Image&Edit
DragonDiffusion
DiffEdit
Ours
Figure 4: Qualitative comparison between our LucidDrag and other methods in object moving.

---

Qualitative comparisons of the object moving task are shown in Fig. Although the comparison methods can precisely recognize and generate objects the target locations, our method shows better performance in image quality benefiting from our novel framework. On the one hand, the quality guidance constrains the distribution of the generated images, avoiding artifacts and unexpected changes during generation. For example, in the case of the water cup on the lower left, the human hands of comparison methods are deformed. Differently from prior work, our method achieves better image fidelity. On the other hand, the intention predicted by our intention reasoner can provide semantic guidance, thus preventing the object from reappearing in the source location. For example, in the case of the doughnut on the lower right, the doughmut of comparison methods still appears in its original position. Conversely, our method successfully moves the doughnut to the target position.
Table 2: Compairisons of object moving.
DragonDiffusion
DiffEdit
Ours
CLIP-score (1)
0.255
0,257
0.260
LMM-score (1) 6.25
6.5
7.25
4.3 Ablation Study
Intention planner LucidDrag introduces the intention reasoner for intention understanding. In one respect, this module enhances semantic-awarg capability during image editing. In another respect, it provides semantic guidance during generation, prompting accurate dragging of handle points to target positions. To substantiate our claims, we present an ablation study of the intention reasoner. We present qualitative experiments in Fig. 5 For example, one possible semantic intention of the case in the first row of Fig. ii to reduce the size of the wheel. When the intention reasoner is removed, i.e., w/o Intention, the model has difficulty in understanding the semantic intention, limiting its dragging ability. Alternatively, our approach provides a strong semantic understanding of intentions, enabling precise dragging of objects to their target positions. Quantitative results presented in Tab Employing the Intention planner yields a 3.20% performance improvement in Mean Distance and 0.61% performance improvement in GScore, further demonstrating the necessity of semantic understanding and the effectiveness of the intention planner.
Quality guidance One of the major differences between our LucidDrag and the previous 50 46 533 is that explicitly introduce quality guidance to improve the image quality of the edited image. To verify the effectiveness of quality guidance, ablation studies are conducted both qualitatively and quantitatively. Specifically, we denote the ablation study of removing quality guidance as w/o Quality. As depicted in Fig. 5] removing quality guidance leads to reduced image quality, evidenced by unexpected changes in the front wheel of the bicycle (the red iron bracket) and artifacts in the spinning top. Ablating reasoned intention reduces the model's 's ability to perceive the target shape. For example, after removing the intention reasoner, there are some distortions of the structure. Additionally, Tab. Tab.3]ddd |eddmoostsstess that ablating quality guidance not only degrades image quality but also impacts edit precision. This may because of the f'act that the unexpected artifacts hinder the dragging performance.
[55]
Table 3: Quantitative result of ablation study.
Full Implementation (Ouss)
wio Insention Reasoner
w/o Quality Guidance
Dietane ())
20.46 = 0.77
68 0.79
22.13 0.64
GScore (t)
7.37
6.76
6.47
Irags & Edit
Fall Eeplementation
wel Quality Galtance
Inteatise Brauner
Figure 5: Visualization of ablation study.

---

5 Conclusion
In this work, we identify the limitations of previous drag editing models in understanding semantic intentions and generating high-quality edited images. In response, we design novel framework called LucidDrag, which involves an intention reasoner to clarify possible intentions and a collaborative guidance sampling mechanism that incorporates explicit semantic guidance and quality guidance. LrcidDrag excels in: i) adequate understanding of semantic intention, improving semantic perception ability and diversity of the generated images; ii) enhanced dragging performance, including improved drag aceuracy and image quality. Extensive results show the efficiency of our approach and the potential for further advancements in semantic-aware drag-based editing.
6 Limitations
Although our method is capable of achieving semantic-awar aware drag-based editing without the need for training, there are some limitations. i) Complex objects are challenging to drag, and unexpected deformation can occur over long distances. This may be attributed to difficulties in comprehending the intricate nature of the object inaccurate object tracking. In future work, we will investigate the potential for further improvements in performance by utilizing more powerful image generation models and incorporating comprehensive intention understanding. ii) Hyperparameters involved in the editing process will affect the editing results. In future work, we intend to utilize LLM as an agent automatically determine model hyperparameters to enhance the editing performance.
Acknowledgement This research is sponsored by National Natural Science Foundation of China (Grant No. 62306041), Beijing Nova Program (Grant No. 2211100002121 106, 20230484488), CCF-DiDi GALA Collaborative Research Funds (Grant No. 202414), and Beijing Municipal Science & Technology Commission (223110000742305))
References
[1] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXi::2333.087 08774, 2023.
[2] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In CVPR, 2022.
[3] Arpit Bansal, Hong-Min Chu, Avi Schwarzschld,, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. CVPR, 2023.
[4] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix(pix: Learning to follow image editing instructions. In CVPR, 2023.
[5] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qic, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023. [6] Minghao Chen, Tro I Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In WACV, 2024.
Wei-L.in in Chiang, Zhuohan Li, 7i Lin, Ying Sheng, Zhanghao Wa, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chathot impressing gpt-4 -0000023330000000mmm00-------- 2023.
with 90%* chatgpt quality. URL:
[8] Gabriele Corso, Yilun Xu, Valentin De Bortoli, Regina Barzilay, and Tommi S Jaakkola. Particle guidance:
non-iid diverse sampling with diffusion models. ICLR, 2023.
[9] Xing Cui. Zekun Li, Peipei Li. Yibo Hu, Hailin Shi, Chunshui Cao, and Zhaofeng He. Chatedit: Towards multi-tur turn interactive facial image editing via dialogue. In EMNLP, 2023.
[10] Xing Cui, Zekun Li, Pei Pei Li. Huaibo Huang, and Zhaofeng He. Instastyle: Inversion noise of a stylized image is secretly a style adviser. arXiv preprint urXiv:2311.15044 2023.
[11] Wenliang Dai. Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang
Li, Pascale N Fung. and Steven Hoi. Instructblip: Towards general general-purpsee vision-language models with instruction tuning. In NeurIPS, 2024.
[12] Prafulla Dhariwal and Alexander Nichol. Diffuusion models beat gans on image synthesis. In NeurIPS, 2021.
[13] Wenkai Dong. Song Xue, Xiaoyue Duan, and Shumin Han. Prompt tuning inversion for text-driven image editing using diffusion models. In /CCV, 2023.
[14] Y Endo. cottrloottll latent transformer for stylegan image layout editing. In Computer Graphics Forum, 2022.
10

---

[15] Dave Epstein. Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. In NeurlPS, 2023.
[16] Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfci Yang, and Zhe Gan. Guiding instruction- based image editing via multimodal large language models. In ICLR, 2024.
[17] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In
ICLR, 2022.
[18] Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, and Peter Wonka. Llm blueprint: Enabling text-to-image generation with complex and detailed prompts. In ICI.R, 2023.
[19] lan Goodfellow, Jean Pouget-Abadin Mehdi Mirza, Bing Xu, David Farrdey Sherjil Ozair, Aaron Courville, and Yoshua Bengio, Generative adversarial networks. CACM, 2020.
[20] Alexandros Graikos, Nikolay Malkin, Nehojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and- play priors. In NewrlPS, 2022.
[21] Bohai Gu, Heng Fan, and Libo Zhang. Two birds, one stone: A unified framework for joint learning of image and video style transfers. In ICCV, 2023.
[22] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without
training. In CVPR, 2023.
[23] Wes Gurnee and Max Tegmark. Language models represent space and time. In ICLR, 2024.
[24] Cheng Han, James Chenhao Liang, Qifan Wang, MAJID RABBANI, Sohail Dianat, Raghuveer Rao, Ying Nian Wu, and Dongfang Liu. Image translation as diffusion visual programmers. In ICLR, 2023. 2023.
[25] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. In ICCV, 2023.
[26] Amir Hertz, Ron Mokady, Jay Tenenhaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-
prompt image editing with cross-attention control. In ICI.R, 2022.
[27] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Demoising diffusion probabilistic models. In NeurlPS, 2020.
[28] Jonathan Ho and Tim Salimans. Classifier fresi diffusion guidance. In NewrIPSW, 2021.
[29] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[30] Junjie Ke, Qifei Wang. Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scalc image quality
transformer. In /CCV, 2021.
[31] Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance score-based diffusion models. In ICML, 2023.
[32] Mingi Kwon, Jacscok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space.
In ICLR, 2023.
[33] Peipei Li, Yibo Hu, Ran He, and Zhenan Sun. Global and local consistent wavelet-domain age synthesis.
IEEE TIFS, 2019.
[34] Peipei Li. Huaibo Huang. Yibo Hu, Xiang Wu, Ran He, and Zhenan Sun. Hierarchical face aging through disentangled latent characteristics. In ECCV, 2020.
[35] Peipei Li, Rui Wang, Huaibo Huang, Ran He, and Zhaofeng He. Pluralistic aging diffusion autoencoder. In ICCV, 2023.
[36] Peipei Li, Xiang Wu, Yibo Hu, Ran He, and Zhenan Sun. M2fpa: A multi-yaw multi tulti-pitch high- -ualityy dataset and benchmark for facial pose analysis. In ICCV, 2019.
[37] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt under- standing of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305. 13655, 2023.
[38] Pengyang Ling, Lin Chen, Pan Zhang, Huaian Chen, and Yi Jin. Freedrag: Point tracking is not you need for interactive point-based image editing. In CVPR, 2024.
[39] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022.
[40] Xuannan Liu, Pei Pei Li, Huaibo Huang, Zekun Li, Xing Cui, Weihong Deng, Zhaofeng Hc, ct al. Fka-owl: Advancing multimodal fake news detection through snowledge-augmented Ivims. In ACM MM, 2024. [41] Xuannan Liu. Zekun Li. Peipei Li, Shuhan Xia, Xing Cui, Linzhi Huang, Huaibo Huang, Weihong Deng. mixed-soures multimodal misinformation detection benchmark for Ivlms. arXiv preprint arXiv:2406.00 087727 2024.
and Zhaofeng He. Mmfakebench:
[42] Grace Luo, Trevor Darrell, Oliver Wang, Dan B Goldman, and Aleksander Holynski. Readout guidance: Learning control from diffusion features. CVPR, 2024.
[43] Jiafeng Mao, Xueting Wang, and Kiyoharu Aizawa. Guided image synthesis via initial image editing in diffusion model. In ACM MM, 2023.
[44] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023.
[45] Chong Mou, Xintao Wang. Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation diffusion models. In ICLR, 2023.
11

---

[46] Chong Mou, Xintao Wang, Jiechong Song. Ying Shan, and Jian Zhang. Diffeditor: Boosting accuracy and fexxibliity on diffusion-based image editing. In CVPR, 2024.
[47]
Hyelin Nam, Gihyun Kwon, Geon Ycong Park, and Jong Chul Ye. Contrastive denoising score for text-guided latent diffusion image editing. In CVPR, 2024.
[48] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mogrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with
text-guided diffusion models. In ICML, 2022.
[49]
OpenAL. Chatgpt. In URL: https://peeaa.cco//bbbe
2022.
[50]
Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIG- GRAPH,, 2023.
[51]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[52]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
[53] Yujun Shi, Chuhui Xue, Jiachun Pan, Wenging Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion:
Harnessing diffusion models for interactive point-based image editing. In CVPR, 2024.
[54] Jascha Dicl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning
using nonequilibrium thermodynamics. In ICML, 2015.
[55] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
[56] Yang Song. Jascha DicksDic,,,,in, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In /CLR, 2020.
[57] Qianrui Teng. Rui Wang. Xing Cui, Peipei Li, and Zhaofeng He. Exploring 3d-aware lifespan face aging via disentangled chape-texture representations. In ICME, 2024.
[58] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Manee-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[59] Shanu Vashishtha, Abhinav Prakash, Lalitesh Morishetti, Kaushiki Nag. Yokila Arora, Sushant Kumar, and Kannan Achan. Chaining text-to- text-to-image and large language model: A novel approach for generating personalized e-commerce banners. arXiv preprint S4Mi::2003305578 2024.
[60] Rui Wang. Hailong Guo, Jiaming Liu, Huaxia Li, Haibo Zhao, Xu Tang. Yao Hu, Hao Tang, and Peipei Li. Stablegarment: Garment-centric generation via stable diffusion. arXiv preprint arXiv:2403. 10783, 2024. [61] Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In ICCV, 2023.
[62] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Fdd Chi, Quoc V Le.. Denny Zhou, et al. Chain-of-thouge prompting elicits reasoning in large language models. In NeurlPS, 2022.
[63] Jinheng Xie. Yuexiang Li. Yawen Huang, Haozhe Liu. Wentian Zhang. Yefeng Zheng. and Mike Zheng Shou. Boxdiff: Text-to-image syathesis with training-fre free box-constrained diffuusion. In ICCV, 2023. [64] Serin Yang, Hyunmin Hwang, and Jong Chul Ye. Zero-shot contrastive loss for text-guided diffusion image style transfer. In ICCV, 2023.
[65] TaeHo Yoon, Kibeom Myoung. Keon Lee, Jaewoong Cho, Albert No and Ernest Ryu. Censored sampling of diffusion models using 3 minutes of human feedback. In NeurlPS, 2024.
[66]
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang. Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any gramularity. ICLR, 2024. Yugian Yuan. Wentong Li, Jian Liu. Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey:
[67]
Pixel understanding with visual instruction tuning. In CVPR, 2024.
[68] Kai Zhang. Lingbo Mo. Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guiie -uuied image editing. In NewrlPS, 2024.
[69] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In ICCV, 2023.
[70] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[71] Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese. Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing.
arXiv preprint (cxXiv:230309611
2023.
[72] Zewei Zhang. Huan Liu, Jun Chen, and Xiangyu Xu. Gooddrag: Towards good practices for drag editing with diffusion models. arXiv preprint arXiv:2404.0r 07206, 2024.
[73] Jing Zhao, Heliang Zheng, Chaoyue Wang, Long Lan, Wanrong Huang, and Wenjing Yang. Null-text guidance in diffusion models is secretly cartoon-style creator. In ACM MM, 2023.
[74] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2044.1059, 2023.
12

---

A
Appendix
This appendix contains additional details for the NeurIPS 2024 submission, titled "Localize, Un- derstand, Collaborate Guide: Semantic-aware Dragging via Intention Reasoner." The appendix is organized as follows:
A.1
Additional Preliminaries of Diffusion Model.
A.2
Algorithm Pipeline of LucidDrag.
A.3
Instruct Prompts for LLM.
A.4
More Comparsions.
A.5
More Analysis.
A.6
Social Impacts.
A.1
Additional Preliminaries of Diffusion Model
DDIM In the inference time of a diffusion model, given an initial noise vector 27. the noise is
gradually removed by sequentially predicting the added noise for T steps. DDIM [55] is one of the efficient denoising approaches that follow a deterministic process, in contrast to the original
stochastic diffusion process:
01-1 24 +
1
EB.
(13)
Ott
at--
where Eg is the estimated noise.
[55 44 refers to technique that aims to transform an input image into a noise vector 2r conditioned on a given prompt or target representation. This process accomplished by reversing the diffusion process, whereby the final noise sample 2T is returned to the initial 20.
DDIM inversion
ac+1
(VaL--
1 a1-1
2++1
2t +
V Cz
E0-
(14)
Classifier-free guidance For text-based image generation using diffusion models, the classifier-fres guidance technique is often employed to address the challenge of amplifying the effect the text condition. To this end, Ho et al. [28] have presented the classifier-free guidance technique, whereby the prediction also performed unconditionally and then extrapolated with the conditioned prediction. Specifically, the estimated noise in Eq. is adjusted follows:
Eo(21, t, y, Ø) c0(2,, Ø) )
(co(2), t, v) t, 2)),
(15)
where @ (""") is the embedding of null text. c0(2,, t, y) represents the conditional predictions. w is the guidance scale parameter.
Classifier guidance The diffusion sampling process can be guided by variety of energy functions, g(2;; 1, v), which are not limited to probabilities derived from a classifier. Such energy functions may comprise, for instance, the approximate energy derived from another model [39], similarity score derived from CLIP model [48] iineiindependent energy in universal guidance 3 bounding box
penalties on attention (6). any attributes of the noisy images.
The combination of this additional guidance with "classifier-free guidance" [28] enables the generation of high-quality text-to-ine -mage samples that also possess low energy according the energy function g:
é (1 s)co(2;tt,)) v) scp(2,; + v/V (2,gg
(16)
where a represents strength of the classifier-free guidance; v is an additional weight for guidance provided by y. As with classifier guidance, we scale by to convert the score function to prediction of ét. Our work contributes by identifying energy functions g that can be used to control the properties of objects and interactions between them.
13

---

A.2 Algorithm Pipeline of LucidDrag
To facilitate the understanding of our LucidDrag, we present the entire algorithm pipeline in Algo- r The intention reasoner employs locator and a reasoner to infer the underlying intentions. Subsequently, collaborative guidance is leveraged to generate images through the integration of semantic guidance, quality guidance, and editing guidance.
Algorithm 1: Proposed LucidDrag
Require:
pre-trained SD> <g; image to be edited zo; editing guidance steps n1. quality guidance steps f2 (n2 < n1).
Intention Reasoner:
(I) Locatize the interested region, O LV P).
(2) Understand the possible intentions, D = LLM(O.C,P).
(3) Sample source prompt and target prompt: P3J
Collaborative Guidance:
8 (1) zg (ncoder(zo) (2) Invert 2g to 2T gud DDIM 1 .P.). Then, build
the memory bank.
(3) Initialize 27 with 27 =quud
10 for / T,
do noise prediction: é, c (Ze, l G, Ctm);
12
if - - then
13
compute Gedit by Eq. 11
14
if - < then
15
compute Iguality by Eq. 8
16
compute energy function by Eq. 12
17
else
set energy function as Pedit
inject gradient guidance by Eq.
20
compute žt-1 by Eq.
21
To = Decoder(2a):
Output: wo
A.3 Instruct Prompts for LLM
Our approach employs the advanced spatial and reasoning capabilities of LLMs infer potential intentions. The specific instructions provided in this work are presented below.
Instruct prompt reasoning potential intentions.
You helpful assistant. Given the original description and human-select drag information (including the description of the start point and the drag direction), reason the intention of the human based on the drag information, then generate refined source prompt and target prompt. The intention of the human is dragging some part of the image to make some deformation (bigger, shorter, longer, open, closed, etc.) or change its posture (looking left, looking right, looking sideways, looking closer, etc.). The source prompt and target prompt should be similar and reflect the difference before and after dragging. Notably, if the difference between the source and target prompt is hard to describe, you can directly set the source prompt and target prompt as the original description. For example:
example
INPUT:
Original description: a red motorcycle. Drag information The start point is the behind of the motorcycle. Direction is [[292, 276], [345,
14

---

275]].
OUTPUT:
Intention: making the motorcycle longer.
Source prompt: short red motorcycle.
Target prompt: long red motorcycle.
example 2:
INPUT:
Original description: a photo of a raccoon. drag information 1: The start point is the nose of squirrel. Direction is [[297, 270], [347,
248]].
OUTPUT:
Intention: make the raccoon look sideways.
Source prompt: photo of raccoon looking forward. Target prompt: a photo of a raccoon looking sideways.
A.4 More Comparisons
Quantitative comparison In the primary paper, Tab. presents the Mean Distance and Gscore, averaged over all samples the DragBench dataset. This supplementary section presents a detailed comparison of the results obtained for each individual category within the DragBench dataset.
The comparison in terms of Mean Distance is presented in Tab. while the comparison terms of GScore is provided in Tab. [5] As demonstrated in Tab. 4 the proposed LucidDrag method achieves the lowest mean distance, thereby confirming the superiority of our technique in accurately dragging the content to the target position.
About the image fidelity metric reported in Tab. 5 LucidDrag approach also demonstrates superior overall average performance. While methods such as DragDiffusion and FreelDaag achieve higher G-scores on some individual classes, this is likely due their of additional model fine-tuning to better fit the input image, which can be time-consuming.
Despite comparable image fidelity, the competing methods deficient in terms of dragging accuracy, as evidenced by the elevated Mean Distance values in Tab. 4 This demonstrates the superiority of the LucidDrag method in maintaining both high-quality image generation and precise content manipulation capabilities.
Table 4: Comparisons of Dragging Accuracy (Mean Distance) on DragBench (L).
Artworks Landscape
City Countryside Animals Head Upper body Full body Interior Other Average
DragDiftusion
30.74
36.55
27.28
43.21
39.22
36.43
39.75
20.56
24.83
39.52
32.60
FreeDrag
31.16
31.92
28.87
33.57
32.12
34.46
19.75
23.27
36.11
DragonDiflusion
26.23
25.42
29.23
34.17
36.13
28.86
48.61
6.97
18.63
37.36
29.16
DiffEditor
24.41
27.52
34.94
38.42
25.62
24.60
24.77
6.81
17.71
35.32
26.01
Ours
19.71
17.76
26.74
26.67
22.74
23.87
20.70
5.56
14.34
26.54
20.46
Table 5: Comparisons Image Fidelity (GScore) on DragBench (t).
Artworks Landscape City Countryside Animals Head Upper body Full body Interior Other
Average
DragDiffusion 6.71
6.19
7.19
6.91
7.10
7,70
7.70
7.30
7.15
6.74
6.94
FreeDrag
7.04
7.24
6.88
6.37
7.06
8.28
8.18
7.33
7.49
7.07
7.15
DragonDiffusion
6.09
6.07
5.47
6.58
6.93
6.42
6.93
7.03
6.09
6.26
Difflditor
6.90
6.58
6.656
6.58
6.47
6.22
6.32
6.52
6.78
6.58
6.62
Ours
7.28
6.95
7.28
6.81
7.14
7.75
7.54
8.05
8.16
7.43
7.37
15

---

painting sunflowers looking up. target: painting
sunflowers looking
source: photo of chair. target: photo of big chair.
source: photo otter with Its hands
down.
target:
octer with its hands up.
source: photo of cat with open eyes. target: photo of cot with closed eyes.
source: photo of a thin glass.
target: photo of fat glass.
source: painting of women with long siting chair, target: An all painting of woman with short scarf siting on chair.
Interior design, sofa with fow back, chair, low plants. target: design, sofa with high back, chair, high plants.
photo boy holding large umbrella. target: photo boy holding small umbrella.
Edage&Edit
DragDiflusion
FreeDrag
DragonnDiflusion
DiffEditor
Ours
Figure 6: More qualitative comparison content dragging.
16

---

Source prompt: cake facing right. Target prompt: "A cuke facing forward."
Source prompe: middle. Target prompt: "A cake the
Source prompt: "A big cake. Target prompt: "A thin cote."
prompt: waterfall the middle." Target prompt: waterfall right."
prompt: "A thin worterfoll." "
straight waterfall, Target prompt: "A winding waterfall,
Target prompe: "A waterfall. "
Source prompt: on left. Target prompt: "A chair on the right."
prompt: thin chatr"" Target prompt: "A wide chair.
chair facing left.
Target prompt: "A chair facing
Source prompt: "A tree on left.
Source prompt: "A thin tree." Target prompt: fat
Source prompt:
"A straight tree." Target prompt: "A winding
Target prompt: "A right."
Image
"A round cusition." Target prompt: "A semicircular
Source prompt:
Source prompt:
big custiion."
Source prompt: "A rushian in the middle.
Target prompt:
Target prompt: cushion top.
Image & Edit
Diverse editing results
Figure 7: LucidDrag allows generating diverse results.
More Qualitative Results More comparison results for the content dragging task are presented in Fig. 6 These results, conjunction with those presented in Fig. (4. of the main paper, substantiate the superiority of our method in terms of both drag-bat based editing performance and image quality maintenance, and also demonstrate the generality of our approach.
Subsequently, we present additional qualitative results that demonstrate the semantic awareness capability of our proposed method. As illustrated in Fig. 7 our approach is capable of generating diverse range of output images that are highly faithful to the intentions that have been deduced.
A.5 More Analysis
Analysis of different LVLMs and LLMs in Intention Reasoner We conduct experiments to examine the performance different LVLMs and LLMs in the Intention Reasoner module. Specif- ically, we utilize Osprey [67] and Ferret [66] for LVLM and Vicuna 771. LLama3 [58), and GPT 3.5 [49] for LLM. We test various combinations, with Osprey+GPT3.5 being the default setting in paper. As shown in Table 6 all combinations outperform the experiment without the Intention Reasoner, confirming its reliability. This reliability stems two-fold: the LVLMs are trained with large-scale point-level labeled data and can easily achieve point-level understanding [67]. Therefore, they can understand the user-given points without further fine-tuning. For the LLMs, state-of-the-art LLMs have been proven possess strong spatial reasoning abilities 23 enabling them to deduce reasonable intentions.
Table 6: Results with different LVLMs and LLMs
Intention Reasoner
Ferret+ Vicuna
Ferrett LLama3
Ferret+ GPT3.5
Osprey+ Vicuna
Osprey+ LLama3
Osprey+
GPT3.5 (Ours)
Mean Distance (L)
23.66
22.49
21.96
20.65
20.84
20.48
20.46
GScore (1)
6.76
7.11
7.35
7.27
7.13
7.37
17

---

Analysis of the confidence probabilities We analyze the confidence probabilities of the intention reasoner. As shown in Fig. [8 the confidence probability reflects the quality of the output text LLM. A higher confidence probability indicates that the intention of the output is more reasonable, leading better editing results.
Confidence
Confidence
Confidence-
Source promptt photo e noxl promot: to pooto o plaeoppree"
Source prompe procs of nol!
Source prompe: phote of tall
Source "A ploco of staneed provapple- "s photo of pincoople acanolig
Target promppt: photo pinetrple."
pheco of planoople."
Input
Output
Output
Output
Output
Figure 8: Analysis of confidence probabilities. A higher confidence probability indicates that the intention of the output is more reasonable, leading to better editing results.
Analysis of the guidance Fig. 9 illustrates the evolution of gradient maps during the drag-base based editing task at different time steps. The first row of the figure depicts the gradient maps produced by the editing guidance, Jeda. while the second row depicts the gradient maps generated by the quality guidance, Iyuakiyy The visualizations presented demonstrate a process of gradual convergence. In particular, as the sampling progresses, the activation range of the gradient maps narrows progressively, gradually converging toward the respective editing areas.
T-40
T-35
T-30
T-25
T-20
Qquality
gelit
Image User Edit
LucidDrag
Figure 9; Visualization of the quality guidance editing guidance.
Fig. 10 presents visual analysis of the influence of the weight of quality guidance Wiuaaiiy on the image editing results. The figure illustrates that increasing the weights can strengthen the importance of their respective energy functions. However, in distortion or artifacts in the image, there is a trade-off between the editing effects and the image quality. Consequently, in our design, have set Cauality le-3 to avoid excessive constraints from the quality guidance, which could otherwise limit the effectiveness the image editing process.
greater degree of editing is more likely to result
lc-4
5e-4
le-3 (default)
5e-3
le-2
Figure 10: Analysis of the quality guidance weight.
18

---

The efficiency of different methods We present the efficiency of different methods in Table 7 Our method has relatively small inference time and comparable memory requirements.
Table 7: Efficiency of different methods.
DragDiffusion FreeDrag
DragonDifflssio
DiffEditor
Ours
Time (s) 4
80 12.8
92
30
35
48
Memory (GB) +
13.1
15.7
15.7
15.8
Bad case As illustrated in Fig. [1 some complex objects are challenging to drag and some unexpected deformation occurs when dragging over long distances. This may be attributed to difficulties in comprehending the intricate nature object or inaccurate object tracking.
SSS
Figure 11: Bad case of our LucidDrag.
A.6 Social Impacts
The emerging LucidDrag technology has promising applications in image editing, content creation, and visual design, inspiring people to create art. However, the use of this technology requires critical evaluation of potential negative consequences, such as the generation of false or misleading content and potential privacy violations. To address these concerns, the integration of robust illegal content identification models viable approach mitigate these risks and ensure the responsible and ethical use of LucidDrag within the creative landscape.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In our abstract and introduction, we propose LucidDrag, a model that en- ables semantie-aware dragging via LLM intention reasoner and enhances the ability to accurately drag the image with collaborative guidance. Extensive experiments in this paper demonstrate remarkable performance in generating high-quality images with semantic perception.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in the main paper and provide bad cases of our methods corresponding with sufficient analysis in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results. Our contribution is that we consider the semantics during drag-based editing and design collaborative guidance to generate high-quality images.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide an algorithm pipeline in Sec. A.2 in the Appendix. Besides, we fully describe the experiment details, including the experimental setup and data set. This ensures that the contributions of the paper can be independently verified by other researchers.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
21

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: Due to the time limitation, the code has not been prepared well. We will release it as soon as possible. Besides, researchers can reproduce our method based on the algorithm in Sec. A.2 of the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Sublic/guides http://uuuuide/eeeuuueeeeeeeee
CoodeSubmissionooll1
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ trrips.cc/publl. httpa/////guuee///deubbbbbibs/e ott.riddee/cooeSSuubsssssisooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed reporting of the training parameters for the quality discriminator, such as optimizer, learning rate, epochs, and batch size, facilitates replication of experimental results. Similarly, the specification of the key parameters of the diffusion process, including the base model, inversion steps, denoising steps, and classifier guidance strength, allows thorough understanding of the implementation and allows comparisons with other diffusion- based approaches.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
22

---

Answer: [Yes]
Justification: We provide the mean and standard deviation of quantitative indicators in Tab. and Tab.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide Experiments Compute Resources in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps:/neeurips.c//// htce//pullc//Ehccsuuutt EthicsCuudeeines
Answer: [Yes]
Justification: Our work and research fully comply with the ethical guidelines established by NeurlPS to ensure that the research process and its results have a responsible and controllable impact society and the environment.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
23

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impacts in Sec. of the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For cxample, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We utilize the safety checker in the diffusion model to avoid generating unsafe iamges.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: These assets are publicly released, properly licensed, and can be used for scientific rescarch.
24

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Although we plan to open source code, our code is not ready yet. We will provide document with our code when is released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
25

---

Answer:
[NA]
Justification:
Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "4AjbE9R0HX", "reviewer": "Reviewer_ND9N", "summary": "The paper introduces a novel framework for semantic-aware drag-based image editing. Specifically, to address the limitations in understanding semantic intentions and generating high-quality edited images, it utilizes an intention reasoner to deduce potential editing intentions and a collaborative guidance sampling mechanism that integrates semantic guidance and quality guidance. Experimental results validate the effectiveness of the proposed method in producing semantically coherent and diverse image editing outcomes.", "questions": "Please refer to the weakness part.", "limitations": "The limitations are discussed in the supplementary material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well-organized and easy to follow.\n\n2.The proposed what-then-how\" paradigm for drag-based editing is novel and sound.\n\n3.The proposed collaborative guidance is also interesting and has shown to be effective. \n\n4.The proposed method is shown to outperform the existing methods on various editing tasks.", "weaknesses": "1.It seems that the used LVLM and LLM models are not finetuned, and I am wondering how different models perform on the task. Are the confidence probabilities reliable or meaningful without any fine-tuning?\n\n2.The efficiency of different methods should be provided for comparison. \n\n3.The limitations are recommended to be added to the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We are encouraged that you find our ''what-then-how'' paradigm and collaborative guidance mechanism to be novel and effective. Below are our responses to your concerns.\n\n> *W1: How different LVLMs and LLMs perform on the task? Are the confidence probabilities reliable or meaningful without any fine-tuning?*\n\n（1）**We conduct experiments to examine the performance of different LVLMs and LLMs in the Intention Reasoner module.** Specifically, we utilize Osprey [r1] and Ferret [r2] for LVLM and Vicuna [r3], LLama3 [r4], and GPT 3.5 [r5] for LLM. We test various combinations, with Osprey+GPT3.5 being the default setting in our paper. As shown in Table r1, all combinations outperform the experiment without the Intention Reasoner, confirming its reliability without fine-tuning. This reliability stems two-fold: the LVLMs are trained with large-scale point-level labeled data and can easily achieve point-level understanding [r1]. Therefore, they can understand the user-given points without further fine-tuning. For the LLMs, state-of-the-art LLMs have been proven to possess strong spatial reasoning abilities [r6], enabling them to deduce reasonable intentions without fine-tuning.\n\n(2) **The confidence probabilities are reliable and meaningful without fine-tuning. We present qualitative analysis in Fig. 12 in Rebuttal PDF.** As discussed above, the LLM's powerful reasoning capabilities guarantee its reliability without fine-tuning. The confidence probability reflects the quality of the output text in LLM. As shown in Fig. 12, a higher confidence probability indicates that the intention of the output is more reasonable, leading to better editing results.\n\nTable r1: Results with different LVLMs and LLMs\n\n|     | w/o Intention Reasoner | Ferret+Vicuna | Ferret+LLama3 | Ferret+GPT3.5 | Osprey+Vicuna | Osprey+LLama3 | Osprey+GPT3.5 (Ours) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Mean Distance （↓） | 23.66 | 22.49 | 21.96 | 20.65 | 20.84 | 20.48 | **20.46** |\n| GScore (↑) | 6.76 | 7.12 | 7.11 | 7.35 | 7.27 | 7.13 | **7.37** |\n\n[r1] Yuan Y, et al. Osprey: Pixel understanding with visual instruction tuning. In CVPR, 2024.\n\n[r2] You H, et al. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.\n\n[r3] Chiang WL, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. In URL: https://vicuna.lmsys.org, 2023.\n\n[r4] Touvron H, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.\n\n[r5] OpenAI. Chatgpt. In URL: https://openai.com/blog/chatgpt, 2022.\n\n[r6] Gurnee W, et al. Language models represent space and time. In ICLR, 2024..\n\n> *W2: The efficiency of different methods.*\n\nWe present the efficiency of different methods in Table r2. Our method has a relatively small inference time and comparable memory requirements.\n\nTable r2: Efficiency of different methods.\n\n|     | DragDiffusion | FreeDrag | DragonDiffusion | DiffEditor | Ours |\n| --- | --- | --- | --- | --- | --- |\n| Time  (s) ↓ | 80  | 92  | 30  | 35  | 48  |\n| Memory (GB) ↓ | 12.8 | 13.1 | 15.7 | 15.7 | 15.8 |\n\n> *W3: The limitations are recommended to be added to the main paper.*\n\nThanks for your valuable suggestion. We will move the limitations section from the supplementary materials to the main paper in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OLFTq4jaci", "reviewer": "Reviewer_7sLK", "summary": "This paper presents a novel framework called LucidDrag for drag-based image editing. Compared to previous methods, LucidDrag first reasons the intention of the draging operation using LLM (GPT 3.5) and provides a semantic guidance for the following editing process. For the better image fidelity, the authors also design a GAN-based discriminator as the quality guidance, obtaining improved results.\n\nThe authors perform sufficient comparison experiments and ablation studies to support their claims and verify the efficiency of their proposed components. And the authors promise to release the codes.", "questions": "Refer to the weakness.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The writing is well except for some minor issues.\n\n2. The authors note the problem that the drag-based editing has the inherently ill-posed nature, which is because multiple editing results may correspond to the same input image and draging conditions. Based on this observation, the authors attempt to introduce additional prompts to provide semantic guidance for better editing results.\n\n3. The comparisons and ablation studies are sufficient to support the claims.", "weaknesses": "1. It's a good idea to introduce additional prompts to help the editing process. However, why using an Intention Reasoner to \"guess\" multiple intentions and then sampling between them? Maybe a better way is that the users provides their true intention and LLMs are just used to normalize the prompts? So, I'm skeptical about the utility of the Intention Reasoner.\n\n2. Some confusions:\n\n- Line 162-163, how do you get reasonable intentions only taking the generated description of the object of interest 'O', the original image caption 'C', and drag points 'P' as input?  Take Fig.2 as an example, 'O' is \"The nose of a woman\", 'C' is \"A woman\", 'P' is the draging points. There is no information about the original image, so how can LLMs output intentions like \"A woman looking up\"?  Where does the \"looking up\" come from?\n\n- Line 173, $z^{gud}_T$ should be $z^{gen}_T$?  or the $z^{gen}_T$ on the left in Fig.2 should be $z^{gud}_T$?   Maybe the notations in Fig. 2 should be modified.\n\n- Fig. 4, LucidGrag needs additional generated prompts as inputs. You should also demonstrate these prompts to help us understand the editing process. BTW, due to sampling generated intentions based on the confidence probabilities (Line 169), the generated results of LucidGrag should be different. Maybe showing different results can also help us to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We are encouraged by your recognition of the value of our work and your acknowledgment that our experiments sufficiently support our claims. Below is our response addressing your concerns.\n\n> *W1: It's a good idea to introduce additional prompts to help the editing process. However, why using an Intention Reasoner to \"guess\" multiple intentions and then sampling between them? Maybe a better way is that the users provides their true intention and LLMs are just used to normalize the prompts?*\n\nThank you for your suggestion. While allowing users to directly provide their true intent and using LLMs to normalize prompts is indeed feasible, especially when users can clearly articulate their needs, the intention reasoner offers several significant advantages:\n\n(1) **Handling vague requests.** Users' requests often lack specificity, such as \"Drag the horse's head to the top right\" in Fig. 1. The intention reasoner can analyze context data to infer multiple potential intentions (\"long necks\" or \"heads up\" or \"closer\"), allowing users to select the most appropriate one for execution.\n\n(2) **Expressing complex needs.** Articulating complex manipulations involving multiple points or objects can be challenging for users. The intention reasoner can generate precise descriptions automatically, ensuring accurate and consistent adjustments, thereby simplifying the user's task.\n\n(3) **Discovering potential needs.** As shown in Fig.1, the intention reasoner can present multiple possibilities, enabling users to discover and explore various editing choices they might find beneficial, thus enhancing the overall experience.\n\n(4) **Reducing cognitive load.** Many users, particularly beginners, may struggle to provide their intentions precisely. The intention reasoner can infer the potential intentions, alleviating the need for detailed instructions and significantly improving operational efficiency.\n\nAdditionally, by leveraging the reasoning ability of LLMs, we can generate a variety of reasonable intentions and produce high-quality results. Generating diverse results is a challenging task [r7]. Our method can be used to construct an image editing dataset with diverse editing outcomes, which may inspire future work.\n\n[r7] Corso G, et al. Particle guidance: Non-IID diverse sampling with diffusion models. In ICLR, 2024.\n\n> *Confusion1: How do you get reasonable intentions only taking the generated description of the object of interest 'O', the original image caption 'C', and drag points 'P' as input?*\n\nThe intention reasoner can produce reasonable results because large models possess in-context learning, space understanding, and reasoning abilities [r8]. Additionally, previous work has demonstrated that LLMs can generate reliable results through these abilities [r9].\n\nFor the case in Fig. 2, we first combine the inputs with detailed task descriptions and in-context examples to ensure the rationality of the output. Then, given the position of the source point and target point, the LLM can comprehend the dragging direction. Finally, combined with the region of interest (\"the nose of a woman\") and the whole image description (\"a woman\"), the LLM can deduce a reasonable intention, e.g. \"looking up\". This result is logical because dragging the face to the upper left may be an attempt to make the person look up. LLM can easily accomplish this kind of logical reasoning [r8, r9].\n\n[r8] Zhang Y, et al. LLM as a mastermind: A survey of strategic reasoning with large language models. arXiv, 2024.\n\n[r9] Lian L, et al. LLM-grounded Video Diffusion Models. In ICLR 2024.\n\n> *Confusion2: Notations of  $ Z_T^{gud} $ and  $Z_T^{gen}$ in Fig. 2 and Line 173.*\n\nThank you for highlighting the difficulty in interpreting the notations of   $Z_T^{gud}$ and  $ Z_T^{gen}$. To clarify the pipeline, the input image is inverted to $ Z_T^{gud}$ and the generation process starts with  $Z_T^{gen}$. In terms of value, we initialize  $Z_T^{gen}$ with $ Z_T^{gud}$, i.e., $Z_T^{gen} =Z_T^{gud}$. We will improve the clarity of these notations in the updated version.\n\n> *Confusion3: Should demonstrate generated prompts in Fig. 4. Suggest to show different results of various generated intentions.*\n\n(1) We present these prompts in Fig. 13 in Rebuttal PDF. We will incorporate the corresponding prompts in the updated version of our paper.\n\n(2) In Fig. 3 of the main paper, we present the results of various intentions **with different editing targets.** We supplement the results of various intentions **with the same editing target** in Fig. 13 in the Rebuttal PDF file. The results show that various text intentions generated by the intention reasoner with the same editing target will result in similar results and have some differences in details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I2QUqKr9NI", "reviewer": "Reviewer_17V7", "summary": "This paper aims to address the limitation of current dragging-based image editing methods that understand the intentions of users. To this end, the proposed method leverages the reasoning ability of LLMs to infer possible intentions, which are used to provide (asymmetric) semantic guidance in editing. Furthermore, a collaborative guidance sampling method that jointly exploits semantic guidance, quality guidance, and editing guidance is proposed. Among these three types of guidance, quality guidance is provided by a self-trained discriminator with an aesthetic score and images generated by a baseline. The experimental results on the DragBench dataset indicate that the proposed method outperforms several existing methods.", "questions": "Q1: In A.8 the limitations section, why the proposed method is training-free as it includes a trainable quality discriminator?\n\nQ2: How is the computational complexity of the proposed method?\n\nQ3: How do we select the optimal one from the $n$ sampled outputs from the LLM?\n\nQ4: Except for the GScore metric proposed by GoodDrag, has any other commonly-used quality metric (e.g., LPIPS) used for evaluation?\n\nQ5: What will the results look like if the predicted intentions and dragging movements are contradictory?\n\nI might change my score depending on the responses to the above questions.", "limitations": "Yes, the authors have discussed the limitations and the potential negative societal impact of the proposed method in the appendix.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. This paper is well-written and the ill-posedness of dragging-based image editing is indeed a problem that should not be ignored.\n\n\n2. The proposed LLM-based reasoner and the collaborative guidance sampling strategy improve the baseline performance.", "weaknesses": "1. Despite the performance gain, there are observable artifacts in the synthesized images as follows:\n* Page 7, Figure 4, the last row, windows are missing.\n* Page 8, Figure 5, 1st row, right column, the shape and the texture of the mailbox are changed.\n* Page 16, Figure 7, 2nd row, wheels are changed; the shape of the target in the second last row cannot be preserved either. \n\n\n2. Utilizing LLMs to infer intentions is interesting. However, it requires further clarification:\n* How can we ensure that the predicted outputs of LLMs (even with the highest confidence) really coincide with the intentions of users?\n* To the best of my knowledge, the selected baselines do not involve semantic priors from LLMs. Hence, the comparison might be unfair. To fully demonstrate the advantages of the proposed method, text/instruction-based methods should be included for comparison.\n* Even with the provided visualization of the ablation study (Fig. 6), it is still unclear how the introduced components affect the results. Sometimes the full model generates artifacts that do not appear in the results of the variants of the proposed method (e.g., the round bottom of the spinning top in Fig. 6 is changed mistakenly by the full model).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback. Below is our response.\n\n> *W1: Despite the performance gain, there are observable artifacts.*\n\nWe acknowledge these artifacts, a common challenge for drag editing [DragDiffusion, Shi et al. [2024]], but they do not diminish the overall effectiveness of our method.\n（1）**Diverse Generation Capabilities:** Our \"what-then-how\" paradigm offers a novel insight, demonstrating significant improvements in the diversity, reasonability of generated intentions, and interpretability of editing outcomes.\n（2）**Comparative Analysis:** Quantitatively, our method achieves superior editing accuracy and image quality compared to others. Qualitatively, as Fig.4 shows, we successfully positioned all objects in the target area, enhancing overall harmony, and reducing distortions.\n（3）**Adaptive adjustment:** Our method navigates the trade-off between editing accuracy and image quality [DiffEdit, Couairon, et al. [2023]]. Users can utilize adaptive controls to prioritize editing precision or visual quality based on their preferences.\n\n> *W2-1: How to ensure that the predicted outputs coincide with user intentions.*\n\nWe use repeated sampling to explore various plausible user intentions, increasing the probability of covering their true intentions [r10]. **For users with clear intentions**, we utilize advanced intention reasoning techniques and repeated sampling to explore different scenarios (e.g., rigid, non-rigid, rotating). This repeated sampling can boost the probability of LLMs' outputs [10] aligning with the user's true intentions. **For users with unclear intentions**, our system automatically generates plausible semantic intentions by analyzing context and inferring implicit needs, reducing the need for explicit user input and providing reasonable intention estimates. The interactive intention reasoner also **allows users to adjust and refine their intentions in real-time**, helping align the generated results with user preferences and evolving needs, ensuring relevance and accuracy.\n\nThrough these integrated strategies, our method effectively covers and aligns with user intentions, delivering high-quality predictive results.\n\n[10] Brown B, et al. Large language monkeys: Scaling inference compute with repeated sampling. arXiv, 2024.\n\n> *W2-2: Comparisons with text/instruction-based methods.*\n\nWe present quantitative results (Table r3) and qualitative results ( Fig. 13 in Rebuttal PDF). The results show that we outperform them by a significant margin. This advantage may be attributed to the fact that drag editing primarily involves spatial manipulation, whereas the training data of existing text/instruction-based methods [r11, r12, r13] do not include samples with spatial position changes [Drag, Pan et al., 2023].\n\nTable r3: Comparisons with text-based methods.\n\n|     | InstructPix2Pix [r11] | MagicBrush [r12] | ZONE [r13] | Ours |\n| --- | --- | --- | --- | --- |\n| Mean Distance （↓） | 52.34 | 55.77 | 56.83 | **20.46** |\n| GScore (↑) | 7.14 | 7.16 | 6.83 | **7.37** |\n\n[r11] Brooks T, et al. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.\n\n[r12] Zhang K, et al. Magicbrush: A manually annotated dataset for instruction-guided image editing. In NeurIPS, 2023.\n\n[r13] Li S, et al. Zone: Zero-shot instruction-guided local editing. In CVPR, 2024.\n\n> *W2-3: How the introduced components affect the results.*\n\n(1) **Effects of the components.** Removing the intention reasoner results in semantically incorrect results (e.g. the shape of the spinning top and wheel). Removing the quality guidance reduces image quality (e.g. artifacts in the spinning top and the red frame of the front wheel)\n\n(2) **Reasonableness of the shape change of the spinning top.** The source and target prompts are \"a photo of a wide spinning top\" and \"a photo of a narrow spinning top.\" Since the red bottom is part of the spinning top and included in the editing area, narrowing it is semantically consistent. Moreover, the collaboration between the intention reasoner and quality guidance results in a more pronounced narrowing effect in the full implementation. If the user wants to preserve the bottom part, they can exclude it from the editing area(see Fig. 15 in the Rebuttal PDF).\n\n> *Q1: The confusion of the expression training-free.*\n\nThanks. We will correct it in the revised version.\n\n> *Q2: The computational complexity.*\n\nOur method has a relatively small inference time and comparable memory requirements.\n\nTable r4: Computational complexity.\n\n|     | DragDiffusion | FreeDrag | DragonDiffusion | DiffEditor | Ours |\n| --- | --- | --- | --- | --- | --- |\n| Time  (s) ↓ | 80  | 92  | 30  | 35  | 48  |\n| Memory (GB) ↓ | 12.8 | 13.1 | 15.7 | 15.7 | 15.8 |\n\n> *Q3: Details of how to select the optimal one from the  LLM outputs.*\n\nWe utilize LLM to infer N times. Each output contains text $d_j$ and corresponding confidence probabilities $P(d_j)$ which reflect the quality of the text. We then sample based on the confidence probabilities. Details will be added to the revised version.\n\n> *Q4: Performance on LPIPS?*\n\nOur method achieves better or comparable performance. However, we would like to point out that as LPIPS is trained on limited images, it may not effectively differentiate the performance of different models [GoodDrag, Zhang et al. [2024b]].\n\nTable r5: Quantitative comparisons.\n\n|     | DragDiffusion | FreeDrag | DragonDiffusion | DiffEditor | Ours |\n| --- | --- | --- | --- | --- | --- |\n| 1-LPIPS (↓) | 0.137 | 0.116 | 0.124 | **0.113** | 0.114 |\n\n> *Q5: What if the predicted intentions and dragging movements are contradictory?*\n\nWe admit that semantic intent will have a side effect if they are contradictory. Fortunately, no contradictions occur in our experiments, which we attribute to the powerful reasoning ability of LLM [r8]. Even if the contradictions do arise, users can reply to the LLM which will incorporate additional information to satisfy the needs [r9].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qi5oBTZf3o", "reviewer": "Reviewer_6UnY", "summary": "This paper proposes a novel insight to transform the “how to drag” issue into a two-step “what-then-how” by introducing an intention reasoner and a collaborative guidance sampling mechanism. They also raise the problem of image quality issues and design quality guidance to enhance performance. Experiments show their superiority in semantic-aware drag-based editing.", "questions": "I believe that, fundamentally, the user's dragging operation has an intuitive single expectation, but the ambiguity of the operation itself leads to an ill-posed one-to-many mapping. Moreover, this mapping is difficult to traverse and may not even include the user's actual expectation among the diverse versions. Therefore, is it possible to further narrow down the range of expected dragging results, or allow the user to explicitly indicate their needs through additional operations? For other detailed questions and suggestions please refer to the major weakness.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The issue of ‘‘inherent ambiguity of semantic intention’’ seems crucial and interesting. And shifting “how to drag” issue into two-step “what-then-how” can improve the controlling effect of the dragging operation. \n2. The paper is well-written and easy to follow.", "weaknesses": "Major：\n1. How were the single experimental results (like Figure 4) selected from the diverse results conforming to the intention? If they were manually chosen, could this lead to unfair comparisons?\n2. Providing the inferred potential intentions, source, and target prompts corresponding to each generated example may help explain the reasons for superior performance. For instance, in the third row of Fig. 5 on the left side, considering that the hand and the dragging operation should not be related to the corresponding prompt, why do other methods struggle with handling the hand information, while the proposed method can manage it effectively?\n\nMinor:\n1. Fig. 5 caption, misspell of DragonDif(f)usion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We are encouraged that you find our ''what-then-how'' paradigm to be novel and effective. Here is our response to address your concerns.\n\n> *W1: How were the single experimental results (like Figure 4) selected from the diverse results conforming to the intention? If they were manually chosen, could this lead to unfair comparisons?*\n\nAs discussed in Section 3.1, the intentions of the single experimental results are selected based on confidence probabilities to make a fair comparison. We will include relevant explanations in the revised version to avoid misunderstandings.\n\n> *W2: Providing the inferred potential intentions, source, and target prompts corresponding to each generated example may help explain the reasons for superior performance. For instance, in the third row of Fig. 5 on the left side, considering that the hand and the dragging operation should not be related to the corresponding prompt, why do other methods struggle with handling the hand information, while the proposed method can manage it effectively?*\n\n(1) Thanks for your suggestion. We present these prompts in Fig. 13 and Fig. 14 in rebuttal PDF. We will incorporate the corresponding prompts in the updated version of our paper.\n\n(2) The intention, source prompt and target prompt of the third row of Fig. 5 are \"move the cup to the top right.\", \"a cup of coffee on the bottom left.\" and \"a cup of coffee on the top right.\", respectively.\n\n- **Semantic guidance focusing more on the cup means less variation in other areas (hands)**. By using semantic-aware prompts, semantic guidance directs the editing process to focus on the cup area, thus avoiding changes to the hand.\n  \n- **Quality guidance guarantees better image quality and avoids hand distortion.** We propose quality guidance, which maintains image quality via a score-based classifier in the editing process. This approach improves overall image quality and prevents hand distortion.\n  \n\n> *W3: Misspelling of DragonDif(f)usion in Fig. 5 caption.*\n\nThank you for pointing out the misspelling error. We will correct it in the updated version.\n\n> *Q1: The user's dragging operation has an intuitive single expectation, but the ambiguity of the operation itself leads to an ill-posed one-to-many mapping. Moreover, this mapping is difficult to traverse and may not even include the user's actual expectation among the diverse versions. Therefore, is it possible to further narrow down the range of expected dragging results, or allow the user to explicitly indicate their needs through additional operations?*\n\n(1) **Covering the user needs.** Our approach explores diverse plausible user intentions through repeated sampling with LLMs, significantly increasing the likelihood of covering the user's request. **For users without clear intentions**, we could present a diverse set of plausible intentions for them to choose from, inspiring new ideas, without requiring further input from the user. **For users with clear intentions**, we consider various scenarios (rigid, non-rigid, rotating, etc.) and use repeated sampling with LLMs to explore diverse possibilities, as outlined by Brown B, et al. [r14]. This approach significantly increases the likelihood of covering the user's actual intention.\n\n(2) **Allowing flexible interaction.** The intention reasoner allows the users to further narrow down the range of expected dragging results. Specifically, the user can input external constraints to the LLM to limit the scope of generated intents. They can also select the desired intent from a variety of generated intents.\n\n(3) **Advantages of the intention reasoner.** It offers significant benefits over user-providing intentions. The intention reasoner can handle vague requests, express complex needs, discover potential needs, and reduce cognitive load.\n\n[r14] Brown B, et al. Large language monkeys: Scaling inference compute with repeated sampling. arXiv, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and insightful comments which have helped improve our paper. We are encouraged that the reviewers found the importance of addressing the ill-posedness of dragging-based image editing which we aim to solve(Reviewer 17V7, 6UnY). We appreciate their positive feedback on the novelty of our 'what-then-how' paradigm (Reviewers ND9N, 6UnY), the effectiveness of our collaborative guidance (Reviewers ND9N, 7sLK, 17V7, 6UnY), and the sufficiency of our experiments in supporting our claims (Reviewers ND9N, 7sLK, 6UnY). We have provided detailed answers to each question below. We hope that our response addresses these concerns.

Meta-review: This paper introduces an intriguing framework called LucidDrag for semantic-aware drag-based image editing. LucidDrag comprises an Intention Reasoner and a Collaborative Guidance Sampling mechanism. The Intention Reasoner leverages both an LVLM and an LLM to infer semantic intentions, which are then used to facilitate semantic-aware editing. The paper received one weak accept (Reviewer ND9N), two borderline accepts (Reviewers 6UnY and 17V7), and one borderline reject (Reviewer 7sLK).

Most reviewers acknowledged that addressing the ill-posedness of drag-based image editing is crucial and should not be ignored. The main concerns raised by the reviewers relate to 1) whether using an LLM to infer intentions during editing is effective and whether it accurately integrates the user’s true intentions (as noted by Reviewers 7sLK, 17V7, and 6UnY), and 2) the experimental evaluations.

Many of the concerns raised by Reviewers ND9N, 17V7, and 6UnY were addressed during the rebuttal, unfortunately Reviewer 7sLK did not have a chance to respond. After carefully reviewing the paper, reviews, and rebuttal, while the overall technical novelty is limited, as no novel core modules are introduced, the framework’s use of LVLM and LLM to infer possible intentions for drag-based image editing is inspiring. Therefore, this paper is recommended for acceptance.
Decision: Accept (poster)