CLIPDrag: Combining Text-based and Drag-based Instructions for Image Editing
Authors: Ziqi Jiang
Zhen Wang
Long Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Computer Vision
Generative Model
Diffusion Model
Image Editing.
Primary area: generative models

Abstract: Precise and flexible image editing remains a fundamental challenge in computer vision. Based on the modified areas, most editing methods can be divided into two main types: global editing and local editing. In this paper, we choose the two most common editing approaches (\ie text-based editing and drag-based editing) and analyze their drawbacks. Specifically, text-based methods often fail to describe the desired modifications precisely, while drag-based methods suffer from ambiguity. To address these issues, we proposed \textbf{CLIPDrag}, a novel image editing method that is the first to combine text and drag signals for precise and ambiguity-free manipulations on diffusion models. To fully leverage these two signals, we treat text signals as global guidance and drag points as local information. Then we introduce a novel global-local motion supervision method to integrate text signals into existing drag-based methods by adapting a pre-trained language-vision model like CLIP. Furthermore, we also address the problem of slow convergence in CLIPDrag by presenting a fast point-tracking method that enforces drag points moving toward correct directions. Extensive experiments demonstrate that CLIPDrag outperforms existing single drag-based methods or text-based methods.

Published as a conference paper at ICLR 2025
CLIPDRAG: COMBINING TEXT-BASED AND DRAG- BASED INSTRUCTIONS FOR IMAGE EDITING
Ziqi Jiang, Zhen Wang, Long Chen The Hong Kong University of Science and Technology (z i anggbl, zwang j @connect ust. hkk longchen sragchen@ust.oo @ust. hk
Latent Opt mization
CLiPDrag
CUPDraa
(No prompt)
PPeeise but mmbguoous)
"The woman is s silllngg"
EEllagge face of the woman."
(a) Drag-based Edit
Model Feetunee
CLiDDrag
CLiPDrag
sculpture is smiling."
(Definite but not precise)
"The sculpture smiling."
"The sculpture
smiling."
(b) Text-based
(c) Text-Drag Edit
Figure 1:
Three different kinds of image editing methods. (a): Drag-based Editing. Users need to click handle points (red) and target points (blue). (b): Text-based Editing. Only the edit prompt is needed to perform the edit. (c): Text-Drag Editing. both drag points and edit prompts are required.
ABSTRACT
Precise and flexible image editing remains a fundamental challenge in computer vision. Based on the modified areas, most editing methods can be divided into main types: global editing and local editing. In this paper, discussed two representative approaches of each type (i.e., text-based editing and drag-based editing. Specifically, we argue that both two directions have their inherent draw- backs: Text-based methods often fail to describe the desired modifications pre- cisely, while drag-based methods suffer from ambiguity. To address these issues, we proposed CLIPDrag, a novel image editing method that is the first try to combine text and drag signals for precise and ambiguity-free manipulations on diffusion models. To fully leverage these two signals, we treat text signals as global guidance and drag points as local information. Then we introduce a novel global-local motion supervision method to integrate text signals into existing drag- based methods (Shi et al., 2024b) by adapting pre-trained language-visite model like CLIP (Radford et al., 2021). Furthermore, slow convergence CLIPDrag by presenting a fast point-tracking method that enforces drag points moving toward correct directions. Extensive experiments demonstrate that CLIPDrag outperforms existing single drag-based methods or text-based methods.
also address the problem of
INTRODUCTION
Recently, notable breakthroughs in diffusion models (Ho et al., 2020; Song et al., 2020a;b), have led to many impressive applications (Meng et al., 2021; Dong et al., 2023; Kumari et al., 2023), Among
"Lonn Chen the corresponding author. Codes: hutps://gituuhcco/7/iiiiiiiim IP..rr....

---

Published as a conference paper at ICLR 2025
them, image editing is recognized as significant area of innovation and has gained enormous at- tention (Kim et al., 2022; Nichol et al., 2021; Sheynin et al., 2024; Valevski et al., 2023). Generally, the goal of this task is to edit realistic images based on various editing instructions. Mainstream methods for image editing can be coarsely categorized into two groups: 1) Global Editing, edits given images by text prompt (Li et al,, 2023; Kawar et al., 2023) or extra image (Zhang al,, 2023a; Epstein et al., 2023) containing global information of the desired modification. Most of them involve finetuning pre-trained diffusion model. 2) Local Editing, mainly consists of drag-based methods (Pan et al., 2023). This framework requires users to click several handle points (handles) and target points (targets) on an image, then perform the semantic editing to move the content of handles to corresponding targets. Typical methods (Mou et al., 2023; Shi et al., 2024b) usually contain motion supervision phase that progressively transfers the features of handles to targets by updating the DDIM inversion latent, and a point tracking phase to track the position of handles by performing a nearest search on neighbor candidate points.
Although the aforementioned methods have gained significant prominence, two drawbacks of them cannot be overlooked. 1) Imprecise Description in Global Editing. Global editing such as text- based image methods (Kawar et al., 2023) is unambiguous but is hard to provide detailed edit in- structions. For example, in Figure ((b), the prompt (i.e. "The sculpture is smiling") tells the model how to edit the image, but it is difficult to provide fine-grained editing information, like the ex- tent of the smile. 2) Ambiguity Issues in Local Editing. Although local editing like brag-based editing methods can perform precise pixcl-Ievel spatial control, they suffer from ambiguity because the same handles and targets can correspond to multiple potential edited results. For example, in Figure (a), there exist two edited results meeting the drag requirements: one enlarging the face, the other making the woman smile. A natural approach to solving the ambiguity problem is to add more point pairs, but it does not work in real practice. This is because the diffusion network is a Markov chain and related errors accumulate as the update continues. Thus when adding more points, usually means more update iterations, which will result in the degradation in image fidelity. In conclusion, local editing is precise but ambiguous while global editing is exactly the opposite.
Since these two kinds of editing are complementary, it is natural to ask: can we combine these two control signals to guide the image editing process? In this way, text signals can serve global infor- mation to reduce ambiguity. Meanwhile, drag signals can act as local control signals. providing more detailed control. However, combining these two signals presents two challenges: 1) How to inte- grate two different kinds of signals efficiently? This is difficult because text- based and drag-bases methods have completely different training strategies. Specifically, most text-based methods (Kim et al., 2022) require finetuning a pre-trained diffusion model to gradually inject the prompt informa- tion. But drag-based methods (Shi et al., 2024b) typically involve freezing the diffusion model and only optimizing the DDIM inversion latent of the given image. Besides, the update in text-based editing often involves the whole denoising timesteps while drag-based approaches only focus on a specific timestep. 2) How to solve the optimization problem and maintain the image quality? Previous drag-based methods are very slow in some situations. Sometimes the handles will stuck at one position for many iterations. In worse situations, they even move in the opposite direction of targets. This phenomenon becomes more serious when adding text signals because combining two different signals usually requires more optimization steps. Thus we need a better approach to optimize the update process while maintaining fidelity as much as possible.
To address these problems, we propose CLIPDrag, the first method to combine text-based and drag- based approaches to achieve precise and flexible image editing (For brevity, we denote this kind of editing as text-drag edit). This approach was built based on the general drag-based diffusion editing framework, which optimizes the DDIM inversion latent of the original image at one specific timestep. Specifically, we propose two modules solve the aforementioned problems after a typical identity finetuning process. 1) Global-Local Motion Supervision (GLMS). The key of GLMS is to utilize the gradient from text and drag signals together for mubbiuitt--liminattee text information, we obtain the global gradient by backward of the global CLIP loss for the latent. Similarly, for drag points, the local gradient is calculated using a similar paradigm in DragDif- fuion (Shi et al., 2024b). Because CLIP guidance methods cannot operate at single fixed step, simply adding these two gradients is ineffective. Specifically, in GLMS, We disentangle the global gradient into two components: Identity Component perpendicular to the local gradient to maintain the image's global structure information, and Edit Component parallel to the local gradient, which will be combined with the local gradient from drag signals to edit the image. By comparing their
Specifically, for
2

---

Published as a conference paper at ICLR 2025
direction, we choose different gradient fusion strategies to either perform the edit or maintain the image identity. 2) Fast Point Tracking (FPT). As we mentioned before, combining drag and text signals will result in slow convergence in the latent optimization. Thus we propose faster tracking method to accelerate the optimization process. FPT is introduced to update the position of handles after each GLMS operation. Specifically, when performing the nearest neighhor search algorithm, FPT masks all candidates that are far away from current targets. Consequently, handles will become closer to targets after each update, and FPT also ensures that the moving trajectory of handles will not be repetitive. It is observed that FPT significantly accelerates the image editing process and improves the image quality to some extent.
Combining these good practices, CLIPDrag achieves high-qualit, results for text-drag editing, Ex- tensive experiments demonstrate the effectiveness of CLIPPrag, outperforming the state-of-the approaches both quantitatively and qualitatively. To summarize, our contributions are as follows:
We have pointed out that previous drag-based and text-based image editing methods have the problems of ambiguity and inaccuracy, respectively.
We propose CLIPDrag, a solution that incorporates text signals into drag-based methods by using text signals as global information.
Extensive experiments demonstrate the superiority and stability of CLIPDrag in text-drag image editing, marking a significant advancement in the field of flexible and precise image editing.
2 RELATED WORK
Diffusion Models. Diffusion models are class of generative models that progressively degrade data by adding noise and then learn the reverse denoising process to generate realistic data. The Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020) was the first to demonstrate that diffusion models could achieve results comparable to offte-of-the-art art GANs (Goodfellow et al., 2020) for unconditional image generation. Subsequent works, such as DDIM (Song et al., 2020a) and ScoreSDE (Song et al., 2020b), refined the theoretical framework and improved performance. Classifier- guided (Dhariwal & Nichol, 2021) and classifier-fres guidance (Ho & Salimans, 2022) methods further explored the potential of diffusion models for conditional generation. Inspired by the impact of large language models, the computer vision community has also trained diffusion models on large datasets achieve commercial-gen grade performance, leading to the development of models such as GLIDE (Nichol et al., 2021), EMU (Sheynin et al., 2024), and Imagic (Kawar et al., 2023). Among these, Stable Diffusion (SD) (Rombach al., 2022) has become particularly popular due to its efficiency. Unlike previous methods, SD projects images into lower-dimensional latent space before adding noise, significantly reducing memory and computational requirements. Build- ing on SD, numerous downstream applications have been proposed, including personalization (Ruiz et al., 2023), style transfer (Zhang et al., 2023b), and inpainting (Lugmayr et al., 2022).
based Image Editing. Unlike text-to-image generation which involves creating an image from scratch (Ho et al., 2022; Dhariwal & Nichol, 2021; Saharia et al., 2022; Gu et al., 2022), text- based image editing is about altering certain areas of given image. DiffCLIP (Kim et al., 2022) leverages contrastive language-image pertaining (CLIP) (Radford et al., 2021) to fine-tune the dif- fusion process, enhancing diffusion models for high-quality zero-shot image editing. SINE (Zhang et al., 2023c) improves editing performance on single images by utilizing a large-scale pre-trained text-to-image model. Paint by Example (Yang et al., 2023) is the first approach employ self- supervised training for more precise control through exemplar guidance. FlexiEdit (Koo et al., 2024) enhances non-rigid editing by reducing the feehffreuencyy components in the target areas. Frompt-to-Promp (Hertz et al., 2022) refines text-based editing by manipulating cross-attention maps during diffusion, enabling more effective modifications. Similarly, MasaCtrl (Cao et al., 2023) achieves complex non-rigid image editing by converting self-attention into mutual attention. InstructPix2Pix (Brooks et al., 2023) integrates a pre-trained large language model with a text-to- image model to generate training data for a conditional diffusion model, facilitating direct image editing from textual prompts. Null-text inversion (Mokady et al., 2023), on the other hand, enhances editing by optimizing default null-text embeddings to achieve desired transformations. While these text-based methods empower users to edit images using natural language, they often lack the preci- sion and explicit control offered by drag-based editing techniques.
3

---

Published as a conference paper at ICLR 2025
"The character
Gg
closing mouth"
Step Result
Step"Result Result
CLIP
6086
SD
0.0.9 00000
6606b
z
Global-Local Gradient Fusion
DDIM Inv
Step Feat
Step f-1
Feat
Step'Feat Feat
Step'Feat
4
Oi
G
h
Patch of h
h
-1
Feat
Patch A
Patch of g!!
Fast Point Tracking
Point of
Patch feature arount target point
Local gradient obtained by motion supervision Global gradient obtained by clip guidance
Feature current handle point
Illustration of our scheme for an intermediate single-step optimization. 2 z'means the opti- mized latent code tt updation. The local gradient and global gradient are calculated by backward- ing the motion supervision loss and CLIP guidance loss with respect to the latent code respectively. Then the Global-Local Gradient Fusion method introduced combine these two information to update the latent code. The new handles are inferred through our fast point tracking method.
Figure 2:
Drag-based Image Editing, Recently, DragGAN (Pan et al., 2023) introduced a novel method for achieving precise spatial control over specific regions of image based on user-provided drag instructions. Building on it, DragDiffusion (Shi et al., 2024b) applied the approach to diffusion mod- els, achieving superior performance and establishing a benchmark for future work. FreeDrag (Ling et al., 2023) reduces the burden of point tracking by introducing two key components: adaptive updates of template features and line search with the backtracking mechanism. StableDrag (Cui et al., 2024) proposes a discriminative point-tracking method to construct stable drag-based edit- ing framework. SDE-Drrag (Meng et al., 2021) offers a straightforward yet powerful approach for point-based content manipulation based on the stochastic differential equation (SDE) framework. LightningDrag (Shi et al., 2024a) enables high-quality image editing by climinating the need for time-consuming latent optimization. Similarly, InstantDrag (Shin al., 2024) achieves fast edit- ing framework by incorporating two optical flow generators. A concurrent work, Region (Lu al., 2024), addresses ambiguity issues by transforming the problem into region- basodd editing through a gradient-free copy-paste operation. Unlike our method, RegionDrag employs a simpler transfer mechanism for achieving edits. Another line of drag-based methods includes DiagonDiffu- sion (Mou et al., 2023) and DiffEditor (Mou et al., 2024), which reformulate the image editing task into a gradient-basee process by defining an energy function that aligns with the desired edit results.
3 APPROACH
Problem Formulation. Given an image I and two text prompts Po and Pe, where Po describes the original image and Pa depicts the edited image. After choosing n handles hu,,., hn and corresponding targets (91.02. .n.. ). a qualified text-drag editing method has to satisfy the following two requirements: 1) Move the feature of point h, to It. while preserving the irrelevant content of the original image. 2) The edited image must align with the edit prompt P...
General Framework: As shown in Figure 2 our method CLIPDrag, consists of three steps:
While use "SD" in the diagram, it may only refer to part of the diffusion model. For example, the "DDIM Iny" only utilizes the VAE component.
4

---

Published as a conference paper at ICLR 2025
1) Identity- Preeetrvinn Finetuning (Sec 3.1): Given the input image I and its description P. We first finetune a pre-trained latent diffusion model Eg using the technique of low-rank adpation (LoRA (Hu et al., 2021)). After the finetuning. at a specific timestep through DDIM inversion. 2, will be optimized for many iterations to achieve the desired edit, and each iteration consists of two following phases.
encode the image into the latent space and obtain the latent 2
2) Global-Local Motion Supervision (Sec 3.2): We denote the latent at t-th step and handle point p; during the k-th iteration 2 and P respectively. At k-th iteration, can calculate global and local gradient vectors for a These two gradients will be combined by our Global-Local Gradient Fusion method, and the final result is used to update 2 to 21 x++1
3) Fast Point Tracking (Sec 3.3): After GLMS, we can update the position of handles with the nearest neighbor search. To accelerate the editing speed while preserving the image fidelity, propose FPT to ensure that new handles are closer to their corresponding targets by filtering all candidates that far away from the targets.
3.1 IDENTITY- PRESEIIY-PRESERVING FINETUING
As analyzed in previous work (Shi et al., 2024b), directly optimizing the latent z, in diffusion-based methods will cause the problem of image fidelity. So finetuning on a pre-trained diffusion model is necessary to encode the features of the image into the U-Net. Specifically, the image and its description prompt Po are used to finetune the diffusion model co through the LoRA method:
Cfe(z,
E., c0t2 + o
(1)
where z is the latent space feature map concerning image I. and AO represent the U-Net (Ron- neberger et al., 2015) and LoRA parameters, 04 and o, are constants pre-defined in the diffusion schedule, e is random noise sampled from distribution (0, (). After the finetuning, we choose a specific timestep and obtain the DDIM inversion latent (Song et al., 2020a) as follows:
VI a₁ co co(e))) + / a+t1 cg (2t)
2++1
var+il
vaz
the latent will optimized in the subsequent process while keeping other parameters frozen.
3.2 GLOBAL-LOCAL MOTION SUPERVISION
This step aims to combine and drag signals, as shown in Figure 3. We will introduce how to process each control information, and then show how to combine them.
CLIP-guidance Gradient. We extract knowl- edge from text signal using a local direction CLIP loss (Kim al., 2022), which aligns the direction between the source image I and gen- erates image i with the direction between orig- inal prompt Po and edit prompt Pe:
g
G
final
Gg
G Ginat
G
(a)
(b)
Global gradient
Edirection
| AT
Local gradient
(3)
- E))),, AT (Pe) (Po). Here Er, E, represents text and image encoder from a pre-trained CLIP model. However, the identity component of P. is canceled out during the calculation of AT, making it difficult to maintain the image iden- tity. Besides, sometimes it is impossible to cal- culate the direction loss because P. is not pro-
where Al = (1((22))
Component of gobal gredient
Gradient update
Filtered candidates
(c)
Target
Candidates
Figure 3: (a) G, is consistent with Gt. (b) Gy contra- diets with Gr. (c) Fast Point Tracking.
For convenience, users opt not to provide description, in which case captioning model, such as GPT-4V will automatically generate prompt.
5

---

Published as a conference paper at ICLR 2025
vided (Zhang et al., 2024). Consequently, we choose the global target loss to extract more informa- tion from edit prompt Pe as follows:
Ex)(((E)). Lgtobaa (I, Po) (E)((E))
(4)
Then we can obtain the global gradient (Gg) from the signal: G2 glotoot Oz Later will explain how to use Gg to maintain the image identity and guide the edit.
Drag-guidance Gradient. We denote the U-Net output feature maps obtained by k-th updated latent of as F( And the gradient from drag signals is obtained by the motion supervision loss Lins which the difference between the features of corresponding targets and handles:
Lma(2
Σ
( sg( sg(Fa( + I1(+ sg(=))) o (1 1)I1 (5)
==1 g)s(pp.)1)
Where r(p {3r, y) || sell S rı, lly vill < ni), M is optional mask, and represents the patch radius. sg(+) represents the stop gradient operation (van, 2017), di = (gi / h is an unit vector from he g:- Thus the local gradient (Gt) can be calculated by: G @Lms/O2f.
Global-Local Gradient Fusion. Now we explain how to incorporate the two gradients in detail. The key motivation of this method is to decompose the process of text-based editing process into two parts: the edit component and the identity component. Specifically, as shown in Figure 3, when the direction of the edit component from G, consistent with G3 (Figure 3())), it means both signals agree with how to update the latent. Thus we use Gj to guide the edit while preserving the structure of the image by the identity component of Gg- When the two edit directions are contradictory (Figure 3(b)), we choose to correct the drag direction using the editing component, inspired by (Zhu et al., 2023). This approach can be formalized as follows:
{ GG + (Gn Gi) Gg.
cos g: G1) > 0, G X Cos Gi) Gg. cos Gi) < O,
Gfimal
where G Ginat means the final gradient to update the latent code and X is
3.3 FAST POINT TRACKING
Although CLIP guidance can relieve the ambiguity problem, it makes the optimization of GLMS more difficult. In drag-based methods, similar optimization issue arises when more point pairs are added. In the previous point tracking strategy, handles sometimes get stuck in one position or move far away from their corresponding targets. This significantly slows down the editing process. To remedy this issue, we add a simple constraint on the point tracking process: when updating the handles through the nearest neighbor search algorithm, only consider the candidate points that are closer to the targets, as shown in Figure 3(c). Our FPT method can be formulated as follows:
h
arg min aES( & dis(o.n.) (dds
[1
- Fxę (=²)]l
where (a, 9i) represents the distance between point a and target gt, r2 the search radius.
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We used Stable Diffusion 1.5 (Rombach et al., 2022) and VTTB-VT (Dosovitskiy et al., 2020) as the base model. For the LoRA finetuning stage, we set the training steps as 80, and the rank as 16 with small learning rate of 0.0005. In the DDIM inversion, we set the inversion strength to 0.7 and the total denoising steps to 50. the Motion supervision, we had a large maximum optimization step of 2000, ensuring handles could reach the targets. The features were extracted from the last layer of the U-Net. The radius for motion supervision (ri) and point tracking (rz) were set to 4 and 12, respectively. The weight À in the Global-Local Gradient Fusion process was 0.7.

---

Published as a conference paper at ICLR 2025
"The dog is looking to the right."
(a)
"The character closes its mouth."
(b)
"The dog turns its head to the right."
(c)
"A photo of smiling woman."
(d)
"A photo of Flat shoes."
(e)
User Input
DragDiff
FreeDrag
DiffCLIP
CLIPDrag
Pa
Figure 4: Comparisons with Drag-based methods (DragDiff, FreeDrag) and Text-based methods (DiffCLIP). Pe is the edited prompts, which are required by CLIPDrag and DiffCLIP.
4.2 TEXT-DRAG EDITING RESULTS
Settings. To show the performance of CLIPDrag we compared both drag- brag-based methods (DragDif- fusion, FreeDrag, RegionDrag, StableDrag, InstantDrag. LightningDrag), and text-based method (DiffCLIPP on text-drag image editing tasks. Specifically, drag-based methods require drag points edit instructions while text-based methods need an edit prompt perform the modification. For CLIPDrag, both editing prompts and drag points are required to perform the edit. All input images from the DRAGBENCH datasets (Shi et al., 2024b).
Results. As illustrated in Figure 4, our method shows better performance over the two different edit- ing frameworks. Due to the limited space, the results of RegionDrag, StableDrag, InstantDrag, and LightningDrag are shown in Appendix A. Compared with text-based methods (DiffCLIP), CLIP- Drag can perform ods (DragDiffusion. FreeDrag), CLIPDrag successfully alleviates the ambiguity problem, as shown in Figure 4(b) This is because former drag-based methods intend perform structural editing like moving or reshaping, instead of semantic editing such as emotional expression modification. It reasonable because moving an object is much easier than changing its morphological characteris- tics. Consequently, these models prefer to choose the shortcut to realize feature alignment in motion supervision, resulting
precise editing control the pixel level. Compared with drag-based meth-
ambiguity problem. Our proposed method effectively solves the issue, by introducing CLIP guidance as the global information to point out correct optimization path.
Besides, former drag-based based approaches cannot guarantee edited image quality when multiple drag point pairs exist. So we also give some examples with multiple point pairs to compare the stability of these methods. As shown in Figure 4(a)(c), these results validate the effectiveness of the two techniques in our method: identity component's guidance to preserve the image quality, and fast point tracking to achieve better drag performance.
More results of CLIPDrag shown in Figure 5. The leftmost three examples verify the effective- ness of our method: by combining the information of drag points and edit prompts (P.). CLIPDrag achieves an edit with high image fidelity and ambiguity. The middle three examples show the

---

Published as a conference paper at ICLR 2025
User Input
CLiPDrag
User Input
CLiPDrag
User Input
CLrag
Pat
Pg: The bulb becoming smaller.
A
photo
photo some flowers
of
Pp: The glasses is becoming bigger.
Pg-A photo some brushas
Pe: The gir turning
left.
The The microphana becoming larger.
Pg: photo of cake.
Pp: The chair becoming higher.
Figure 5: Some examples of CLIPDrag. For each input, both an edit prompt and drag points are required. Pe and Po represent the edit prompts and original prompts respectively.
312
Musintani Hostton Maps
OuDrg
tia
User Input
DragDiff
Ours(CLiPDragl
Mags
(a) Comparsion with DragDiff
Quantitative analysis on DRAGBENCH
Figure 6: (a): Examples of CLIPDrag and DragDiff on drag-based edit. Since text information is needed in the editing process of DragDiff, we use the finetuning prompt (Pa) as the editing prompt (P.) in CLIPDrag for faimess. (b): The quantitative experiment on the DRAGBENCH dataset.
situations when users do want consider the ambiguity or find difficult to describe the desired edit. And we found CLIP also works well when the edit prompt is replaced with the original prompt (P.). The rightmost three examples show results when adding mask.
4.3 DRAG-BASED EDITING RESULTS
Settings. Since our method is based on general drag-based frameworks, we also explored the perfor- mance of CLIPDrag in pure drag-bases editing tasks. Specifically, we replaced the editing prompt (Pa) with the corresponding original prompt (P.). This ensures that no extra text information will be introduced. We compared our CLIPDrag with DragDiffusion, FastDrag, and StableDrag on the DRAGBENCH benchmark with five different max iteration step settings. To evaluate the image fidelity, we reported the average 1-LIPIS score (IF). Besides, Mean distance (MD) was calculated to show the distance between the final handles and targets (Lower MD represents better drag per- formance). Due to the limited space, the visual results of StableDrag and FastDrag are included in Appendix E.
8

---

Published as a conference paper at ICLR 2025
(ii)
(iil)
Input
Gradient Adding
User Input
photo of a sculpture."
"The sculpture raising his head."
Our
(a) Ablation on gradient fusion
(b) Ablation on text signal
Figure 7: (a): Ablation study on different gradient fusion strategies. "Gradient Adding" represents simply adding the global and local gradient, and "ProGrad" the method proposed (Zhu al., 2023). The difference between ProGrad and ours is that we utilize the identity component to main- tain image fidelity. (b): Ablation on text signals. for the same drag instruction (e.g. (i)), different results are obtained with different edit prompts ((i) & (iii)).
Loas
(a)
(b)
(c)
(d)
Figure Ablation study on two kinds of point tracking strategies: original point tracking in DragDiff (PT) and our fast point tracking (FPT). for the same drag instruction (a), (b)(c) the results of PT and FPT, respectively. (d)(c) are the corresponding trajectories the handle point.
Results. Quantitative results are shown in Figure 6(b). In this figure, the z-axis represents the max iteration steps and the y-axis represents the IF and MD metrics respectively. As can be seen, most
CLIPDrag has better performance than DragDiff, FastDrag, and StableDrag- Specifically, max iteration steps settings, CLiPDrag has higher IF and lower MD. This means handle points are closer the targets in CLiPDrag, in the meanwhile, the edited image quality is preserved or even improved (Figure 6(a)).
4.4 ABLATION STUDY
Ablation on Text Signals. We performed ablation studies to clarify the effect of text signals in CLIPDrag by using different edit prompts and keeping the drag points unchanged.
Results. As illustrated in Figure 7(b), the ambiguity can be alleviated in CLIPDrag. For example, the drag instruction (i) corresponds to at least two different edit directions: one moving up the sculpture (ii), the other raising his head (iii.. By giving different text signals we can choose different edit paths. We also observed that when no extra text information is given, CLIPDrag tends to align handle and target features by simple position translation (ii), instead of semantic edit (iii).
Ablation on the Global-Local Gradient Fusion. We studied the effect of different methods to combine text and drag signals. We ran this ablation experiment with three different strategies global- local gradient fusion: adding two gradients together, ProGrad, and our method.
Results. As shown in Figure 7(a), adding two gradients cannot alleviate the ambiguity problem because the CLIP guidance does not take effect on the single effect, as illustrated in DiffCLIP. The ProGrad method can relieve the ambiguity problem but performs badly in maintaining the quality

---

Published as a conference paper at ICLR 2025
(a) Input
(b) 10-th Step
(c) 20-th Step
(d) 30-th Step
(c) 40-th Step
Figure 9: Further analysis on Point Tracking strategics. For the same input, intermediate edit results visualized at multiple optimization steps.
of edited images. This is because it ignores the identity component in text signals, which contains information about the image's structure. By contrast, our proposed method effectively fuse the two signals, not only relieving the ambiguity but also maintaining the image fidelity to some degree.
Ablation on Different Point Tracking Strategies. Finally, We showed the effect of different point tracking strategies. We based this experiment on our CLIPDrag method while replacing the fast point tracking with the normal one in DragDiff (Shi et al.. 2024b). Besides, make the result more convincing, we tried to make edited images from these two methods similar by adjusting the random seed and learning rate, while keeping other parameters like patch radius unchanged.
Results. As shown in Figure 8, when achieving similar editing results, FPT effectively reduces the optimization iterations consumed. From the moving trajectory in Figure 8(d), we found that handles could get stuck at one point or move in circles in previous methods. Instead, handles move closer and closer to the targets in our FPT strategy, thus speeding up the editing process. Figure 8(c) shows that the FPT method does not harm the image quality. To further show the effect of FPT, we give another example and their intermediate results, shown Figure 9c)). At the 20th iteration, the FPT strategy began to perform semantic editing, while the PT method was still in the stage of identity-preserving.
CONCLUSION
In this work, we tackled the ongoing challenge of achieving precise and flexible image editing within the field of computer vision. We observed that existing text-based methods often lack the precision for specific modifications, while drag-based techniques are prone to ambiguity. To overcome these challenges, we introduced CLIPDrag, a pioneering method that uniquely integrates text and drag sig- nals enable accurate and unambiguous manipulations. We enhanced existing drag-based methods by treating text features as global guidance and drag points as local cues. Our novel global-loce gra- dient fusion method further optimized the editing process during motion supervision. Additionally, to address the issue of slow convergence in CLIPDrag, we developed an FPT method that efficiently guides handle points toward their target positions. Extensive experiments clearly demonstrated that CLIPDrag significantly outperforms existing methods that rely solely on drag text-based inputs.
ACKNOWLEDGMENT
This work supported by RGC Early Career Scheme (26208924), National Natural Science Foun- dation of China Young Scholar Fund (62402408), Huawei Gift Fund, and HKUST Sports Science and Technology Research Grant (SSTRG24EG04)
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Neural discrete representation learning. In Advances in neural information processing systems, volume 30, 2017.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: funing-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22520 22570, 2023.
Yutao Cui, Xiaotong Zhao, Guozhen Zhang, Shengming Cao, Kai Ma, and Limin Wang. Stabledrag: Stable dragging for point-based image editing. arXiv preprint vrXiv:2033.04437, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In Ad- vances neural information processing systems, volume 34, pp. 9780-8794, 2021.
Wenkai Dong. Song Xue, Xiaoyue Duan, and Shumin Han. Prompt tuning inversion for text-driven image editing using diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1430-7440, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 10X1::2010.11929, 2020.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self- guidance for controllable image generation. Advances in Neural Information Processing Systems, 36:16222-16239, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. pp. 139-144, 2020.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to- text-to-image image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10696-10706, 2022.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- -Or. Prompt-to-pree to-mromptt image editing with cross attention control. arXiv preprint 2XXv:2208.01626, 2022.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in neural information processing systems, volume 33, pp. 6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00 09685, 2021.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2426-2435, 2022.

---

Published as a conference paper at ICLR 2025
Gwanhyeong Koo, Sunjae Yoon, Ji Woo Hong. and Chang D Yoo. Flexiedit: Frequency-awarc latent refinement for enhanced non-rigid editing. arXiv preprint arXiv:2407.17850 2024.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1941, 2023.
Senmao Li, Joost van de Weijer, Taihang Hu, Fahad Shahbaz Khan, Qibin Hou, Yaxing Wang, and Jian Yang. Stylediffusion: Prompt-embeddee inversion for text-based editing. arXiv preprint arXiv:2303.15649 2023.
Pengyang Ling, Lin Chen, Pan Zhang, Huaian Chen, and Yi Jin. Freedrag: Point tracking is not you need for interactive point-based image editing. arXiv preprint arXiv:2307.046: 0466444 2023.
Jingyi Lu, Xinghui Li, and Kai Han. Regiondrag: Fast region-based image editing with diffusion models. arXiv preprint arXiv:2407.18247 2024.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.
Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11461-11471, 2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073: 2021.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation diffusion models. arXiv preprint arXiv:2307.02421, 2023.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Diffeditor: Boosting aceu- racy and flexibility on diffusion-based image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. -488-8497, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.107411 2021.
Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image mani- fold. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10660 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings. part III 18, pp.
Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning rext-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500- 22510, 2023.
12

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding, 2022.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8871- 8879, 2029.
Yujun Shi, Jun Hao Liew, Hanshu Yan, Vincent Y. Tan, and Jiashi Feng. Lightningdrag: Lightning fast and accurate drag-bast based image editing emerging from videos, 2024a.
Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point- image edit- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8839-8849, 2024b.
Joonghyuk Shin, Daehyeon Choi, and Jaesik Park. IrstantDrag: Improving Interactivity in Drag- based Image Editing. In ACM SIGGRAPH Asia 2024 Conference Proceedings, pp. 1-10, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song. Jascha Sohl-Dickster Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020b.
Dani Valevski, Matan Kalman, Eyal Molad, Eyal Segalis, Yossi Matias, and Yaniv Leviathan. Uni- tune: Text-driven image editing by fine tuning a diffusion model on single image. ACM Trans- actions on Graphics (TOG), pp. 1-10, 2023.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18381- 18391, 2023.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 3836-3847, 2023a.
Yuxin Zhang. Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Chang- sheng Xu. inversion-based style transfer with diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10146-10156, 2023b.
Zewei Zhang, Huan Liu, Jun Chen, and Xiangyu Xu. Gooddrag: Towards good practices for drag editing with diffusion models. arXiv preprint Xixx:2444.007006 2024.
Zhixing Zhang, Ligong Han, Amab Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6027-6037, 2023c.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15659-15669, 2023.
APPENDIX
The appendix is organized as follows:
1. In See. A, we show comparisons with more drag-based methods on text-drag edit. 2. In Sec. B, show four more examples in the Figure 1.
13

---

Published as a conference paper at ICLR 2025
3. In Sec. C, show results when apply text and drag guidance sequentially
4. In Sec. D, we show examples when text and drag signals conflict.
5. In Sec. E, show examples of StableDrag, FreeDrag drag-based edit.
6. In Sec. F, we show the inference time comparisons.
A MORE RESULTS OF TEXT-DRAG EDIT.
"The dog looking to the right."
(a)
"The character closes its mouth."
(b)
"The dog turns head to the right."
(c)
"A photo of smiling woman."
(d)
"A photo of Flat shoes."
(e)
User Input
InstantDrag
LightningDrag
RegionDrag
StableDrag
CLIPDrag
P
Figure 10: Results of four drag-based methods, including InstantDrag, LightningDrag, RegionDrag, StableDrag.
To better show the superiority of CLIPDrag compared to traditional drag-based methods, show the results four other drag-based methoossStableDrag LightningDrag, RegionDrag, InstantDrag), shown in Figure 10.
Compared with these methods, CLIPDrag successfully alleviates the ambiguity problem and better maintains the identity as shown in Figure 10. This is because former drag-based methods intend to perform structural editing like moving or reshaping, instead semantic editing such as emotional expression modification. is reasonable because moving object is much easier than changing its morphological characteristics. Consequently, these models prefer to choose the shortcut to realize feature alignment in motion supervision, resulting in the ambiguity problem. Our proposed method effectively solves the issue, by introducing CLIP guidance as the global information to point out correct optimization path.
B EXPLAINATION OF MOTIVATION
To demonstrate our motivation more clearly, we added four results for the second example in Figure 1. Specifically, we provide examples with and without drag edit the following examples. "The sculpture is smiling and not showing his teeth." " and "The sculpture is smiling and not raising his head", shown in Figure 11.
14

---

Published as a conference paper at ICLR 2025
CLIPDrag
DiffCLIP
"The sculpture is smiling and not showing his teeth."
"The sculpture is smiling and not showing his teeth."
CLiPDrag
DiffCLIP
"The sculpture is smiling and not raising his head."
"The sculpture is smiling and not raising his head."
Figure 11: Results of two text prompts with and without drag edit.
CLIPDrag
"The woman is smiling."
Drag Only
Text Only
(No text prompt)
"The woman is smiling."
Text Only
Drag Only
"The woman smiling."
(No text prompt)
Figure Results when the text guidance and drag guidance are applied sequentially.
C APPLY GUIDANCE SEQUENTIALLY
As shown in Figure 12, we demonstrate why applying the two types of guidance sequentially is not considered in our paper. Actually, if we were to apply drag- based editing first, the optimization direction of the latent could incorrect, meaning that the ambiguity problem would occur before the text guidance is applied. Additionally, if text-based editing were applied after the drag operation, the position of the target points would be altered. As result, after the text-based edit, the final positions of the handles would no longer align with the targets, which contradicts the principle of drag-base based methods. Instead, when text-based editing is applied first, the position of the handle points is altered. This alteration mislead the subsequent drag operation.
15

---

Published as a conference paper at ICLR 2025
D CONFLICTING INSTRUCTIONS
"Make the heel of the shoes higher"
À = 0.05
A =5
X =0
Figure 13: Results when the text signals contradict with drag points. is hyper-parameter in Equation (6) to control the strength of text guidance.
While our primary motivation is to use the text prompt to complement the local drag edit ensuring that these two signals are consistent in most cases-Wa also explored scenarios where these guidance signals conflict. Based on our experiments shown in Figure 13, we observed two types of potential editing outcomes:
Ambiguity or Neutralization at Moderate Text Strength (A S 10). When the strength of the text signal is moderate, ambiguity arise if the guidance fails provide accurate information. In other cases, the text signal may counteract the drag operation, effectively neutralizing its effect. For instance, the drag instruction combined with the prompt "Make the heel the shoes higher" might yield result akin to "Make the heel not so high."
Implausible Results High Text Strength(A 100). When the text guidance excessively strong, it overwhelms the denoising process, making it difficult to handle the perturbation. This can result implausible or unrealistic edits.
E MORE RESULTS OF DRAG-BASED EDIT.
Settings. Since our method based on general drag-based frameworks, we explored the perfor- mance of CLIPDrag in pure drag-based editing tasks. Specifically, replaced the editing prompt (Pe) with the corresponding original prompt (Po). This ensures that no extra text information will be introduced. We compared our CLIPDrag with DragDifusion, FastDrag and StableDrag DRAGBENCH benchmark with five different max iteration step settings. To evaluate the image fidelity, we reported the average I-LIPIS score (IF). Besides, Mean distance (MD) was calculated to show the distance between the final handles and targets (Lower MD represents better drag perfor- mance).
the
Results. As can be seen in Table (1)(2,, CLIPDrag has better performance than DragDiff, FastDrag and StahleDraa(thhe
(00,berr10,,,,00.... 160) means the maximum iterations). Specifically, on most max iteration steps settings, CLIPDrag has higher IF and lower MD. This means handle points are closer to the targets in CLIPDrag, in the meanwhile, the edited image quality is preserved or even improved, shown in Figure 14.
Method
10
20
40
80
160
Ours
49.5
45.1
39.2
35.8
32.3
DragDiff
51.3
50.6
42.9
38.8
35.1
StableDrag
50.8
48.8
42.3
39.0
34.8
FastDrag
52.1
51.1
44.4
41.9
36.9
Table I: Mean Distance results.
16

---

Published as a conference paper at ICLR 2025
User Input
StableDrag
FastDraR
Ours(CLIPDrag)
Figure 14: Results of Stable and FastDrag on pure drag-based edit.
Method
10
20
40
80
160
Ours
0.95
0.94
0.93
0.90
0.88
DragDiff
0.95
0.93
0.90
0.87
0.85
StableDrag
0.95
0.90
0.88
0.84
0.81
Fast
0.95
0.93
0.93
0.89
0.83
Table 2: Image Fedlity results.
Method
DragDiff 80.3s
FreeDrag 69.4s
FastDrag 75.58
SDE-Drag
StableDrag
CLIPDrag(Ours)
Time
70.0s
72.3s
47.8s
Table 3: Comparisons of inference time.
F INFERENCE TIME COMPARISONS
In this section, report the average inference time of CLIPDrag, DragDiff, SDB-Drag and Free- Drag(the result is calculated on single 3090 GPU by averaging over 100 examples sampled from the DragBench. As shown in the Table our method is significantly faster than previous works. This is because the inference time is directly correlated with the number of optimization iterations. In CLIPDrag, the text guidance helps to indicate the correct optimization direction, while our FPT strategy prevents handles from moving in the wrong direction or forming loops. Both of these fac- tors reduce the number iterations required to move the handle points to their target positions, resulting in faster editing speeds.
17


Reviews:
[{"review_id": "dONzA6rWKz", "reviewer": "Reviewer_RgXk", "summary": "This paper introduces CLIPDrag, a novel image editing approach that integrates both text-based and drag-based controls to achieve more precise and flexible edits. Traditional text-based editing provides general guidance but often lacks specificity, while drag-based editing offers local control but can be ambiguous without context. CLIPDrag addresses these issues by using text as global guidance for overall image context and drag points for fine-grained, localized control. The model leverages a global-local motion supervision (GLMS) system and a fast point-tracking (FPT) method to streamline and accelerate the editing process. The paper is well written and easy to understand, the paper has comprehensive experimental results which show CLIPDrag outperforms both traditional drag- and text-based methods in accuracy and image fidelity. The detailed ablations make the hypothesis clear. The paper presents an interesting path for image editing and is theoretically grounded which should be shared within the community", "questions": "1. What are some common failure cases for the editing, especially if the text and local edits conflict. \n2. How are the number of iterations for denoising fixed for drag editing and how do the impact change with fewer to larger iterations. \n3. One of example is shown to incorporate masks for editing, can it be explained how masks are incorporated in this framework ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novel Approach to combine local and global gradient: Building on text inversion methods to combine text and drag signals, CLIPDrag enables pixel-level control, offering both specific and contextually aware edits.\n2. Efficient Convergence: The fast point-tracking method improves the editing process by guiding handle points toward their target positions faster.\n3. Extensive Ablations: The paper has ablations for all different components such as point tracking, GLMS and controls with edit and text showing clear performance gain. \n4. Qualitative Results: The papers presents representative set of results allowing easy intuition and help with clarity of the paper.", "weaknesses": "1. The need for identity preservation beyond citing DragDiffusion is not shared, given the improvement of base models, the intuition behind it is lacking. \n2. Gradient accumulation is discussed assuming the latent code is continuous, formulating why the gradient manipulation will still lead to plausible images is unclear. \n3. Assumption around using nearest neighbors in FPT moving monotonically towards target is not explained, given the optimization is highly non linear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YUYSzjLUXv", "reviewer": "Reviewer_R2Bm", "summary": "This paper proposes a Text-Drag Editing framework to address text-based and drag-based editing limitations. To achieve this, the authors introduce global-local motion supervision that integrates the semantic aspects of text with drag guidance. They utilize a novel approach of gradient fusion, combining gradients from text and drag conditioning based on their directional alignment to provide a unified gradient.", "questions": "Please provide examples with and without drag edit the following examples. \"The sculpture is smiling and not showing his teeth.\" and \"The sculpture is smiling and not raising his head\".", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "For the first time, the paper provides an algorithm that integrates text-guided editing with drag-guided editing. The proposed editing algorithm attempts to provide more precise global editing and reduce ambiguity in local editing. The independent guidance or supervision of text and drag is combined interestingly by disentangling the global gradient that is perpendicular and parallel to the local gradient.", "weaknesses": "1. Lack of Comprehensive Review of Diffusion-Based Image Editing Literature:\nThe paper does not provide an adequate overview of diffusion-based image editing methods. A more thorough review of recent approaches in diffusion-based image editing is necessary to strengthen its background and situate the proposed method within the broader field. Specifically, the authors should consider discussing recent methods, such as \nSINE: SINgle Image Editing With Text-to-Image Diffusion Models (Zhang et al., CVPR 2023), \nPaint by Example: Exemplar-based Image Editing with Diffusion Models (Yang et al., CVPR 2023), \nFlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editing (Koo et al., ECCV 2024), \nand RegionDrag: Fast Region-Based Image Editing with Diffusion Models (Lu et al., ECCV 2024). \nIncorporating these examples will provide a more robust foundation and context for the reader, enabling a clearer understanding of how the current approach builds upon or diverges from existing work.\n\n2. Unconvincing Example in Figure 1:\nThe example provided in Figure 1 does not convincingly illustrate the motivation of the study. The intention seems to highlight the limitations of drag-based and text-based editing approaches, yet the figure only demonstrates an instance where drag-based editing is ineffective. A more persuasive example might involve a scenario where drag-based editing produces a subtle change—such as adjusting a subject's smile—which could then be further refined by the proposed text-drag editing method to achieve a more detailed, natural effect. This change would clarify the benefits of text-drag editing over existing methods.\n\nAdditionally, the similarity between the proposed method's results and traditional drag-based editing in Figure 1 and the statute example raises questions about the added benefit of the proposed approach. If these similarities are not intentional, a different example or refinement of the illustrations might better demonstrate the unique advantages of the proposed method.\n\n3. Handling of Distinct Effect Regions in Text-Based and Drag-Based Editing\nThe paper does not adequately explain how it manages distinct effect regions associated with text-based and drag-based editing despite these methods likely targeting different areas in an image. Clarifying how these regions are defined, integrated, or adjusted during editing would provide more specificity and improve understanding of the algorithm's functionality. This discussion is crucial to distinguish the contribution of the combined editing approach.\n\n4. Suggested Comparative Experiments for Method Validation\nComparative experiments should include scenarios where text-based editing is applied after drag-based editing and vice versa to illustrate the proposed method's effectiveness better. This comparison would help demonstrate the practical advantage of combining both methods in the proposed approach and establish whether there are meaningful improvements when they are applied sequentially.\n\n5. Limited Novelty in Gradient Combination Approach\nThe novelty presented in Equation (6), which combines the two editing approaches by decomposing one gradient into the other perpendicular component and then summing them, seems linear, and it is conceivable that a non-linear combination may provide a more effective result.  Including alternative approaches as comparative experiments would strengthen the paper's case for its approach or help contextualize its performance relative to existing methods.\n\nThe paper introduces a combined text-drag editing approach but lacks a comprehensive literature review, convincing examples, clarity regarding region specificity, and evidence of sufficient novelty. Addressing these areas would help elevate the study’s contributions and clarify its position within diffusion-based image editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uH5FT6R8xm", "reviewer": "Reviewer_qFtD", "summary": "CLIPDrag combines text and drag-based controls to improve image editing, using text for broad guidance and drag points for precise adjustments. The author introduces Global-Local Motion Supervision, which combines gradients from both text and drag inputs, and Fast Point Tracking to speed up convergence. This method eliminates common issues like vagueness in text-only edits and ambiguity in drag-only edits.", "questions": "The comparisons in this paper in insufficient, why only DragDiff is compared to in this paper? More comparisons should be added.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe motivation is clear and effective, combining text and drag editing to leverage the strengths of both approaches, achieving more precise edits.\n2.\tThe Global-Local Gradient Fusion method is innovative, merging global text and local drag gradients to enhance editing quality, with experiments showing notable improvements in performance.", "weaknesses": "1.\tThe illustration in Figure 2 is unclear in terms of workflow. If CLIP guidance is applied, the latent space should ideally be converted to the pixel domain to align with CLIP’s processing. However, the diagram uses SD rather than a VAE.\n2.\tCLIPDrag lacks comprehensive quantitative comparisons with other methods in image editing. The current evaluation only includes DragDiff in Figure 6, which is insufficient.\n3.\tThe ablation study also lacks more detailed quantitative comparisons. In Figure 8, the visual differences between (b) and (c) are subtle, making it hard to discern the impact of changes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J6z7euTCQs", "reviewer": "Reviewer_CxEc", "summary": "This manuscript introduces CLIPDrag, a novel method that integrates text-based and drag-based signals for image editing, leveraging both for precise control and reduced ambiguity. The method utilizes Global-Local Motion Supervision (GLMS) and Fast Point Tracking (FPT) to enhance the image editing process, aiming to outperform existing methods by combining the strengths of both editing approaches.", "questions": "In the context of drag-based editing where maintaining the original identity of objects while manipulating specific features is a major challenge, your manuscript suggests the integration of text-based inputs to guide the editing process. Could you elaborate on how the addition of text signals specifically contributes to preserving the object's original identity during the edit? Additionally, are there specific conditions or types of text prompts that particularly enhance this preservation aspect within the CLIPDrag framework?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Innovative Integration**: The paper presents a compelling approach by combining text and drag inputs to guide image editing. This dual-input strategy addresses the limitations of each method when used independently, potentially offering more controlled and precise edits.\n2. **Technical Depth**: The introduction of GLMS shows a deep understanding of the challenges in image editing, particularly in handling the complexities associated with combining different types of editing signals.\n3. **Experimental Validation**: Extensive experiments, including ablation studies, demonstrate the effectiveness of CLIPDrag against state-of-the-art methods. The results are well-presented and support the claims of improved performance in terms of both precision and ambiguity resolution.", "weaknesses": "1. **Novelty of FPT**: The paper should acknowledge that searching for handle points along the path from handle points to targets has been previously explored in methods like DragGAN and DragDiffusion. To clarify the unique contributions of FPT, the authors should provide side-by-side comparisons of point searching strategies, highlighting any improvements or distinctions in their approach.\n\n2. **Comprehensive Comparisons**: While the paper compares CLIPDrag with some existing methods, it would benefit from more extensive comparisons or discussions with recent techniques such as InstantDrag, LightningDrag, StableDrag, and RegionDrag. Although these methods may use different training approaches or inputs, incorporating their text-supervision signals could demonstrate CLIPDrag's ability to address ambiguities present in these methods, showcasing its generalizability. Additionally, these methods should be thoroughly discussed in the related work section to provide a more complete context.\n\n3. **Performance Metrics**: The paper should include a discussion or report on inference time comparisons. This information is crucial for understanding the practical applicability of CLIPDrag in real-world scenarios and how it compares to other methods in terms of computational efficiency.\n\n4. **User Input Optimization**: While the text prompt is provided in DragBench, it's worth noting that the original DragGAN paper did not require text input. The additional text prompt in CLIPDrag may increase user effort. To address this, the authors could explore incorporating vision-language models like GPT-4V to automatically interpret the input image (as shown in the first column of Figure 4). This approach could significantly reduce user burden while maintaining the benefits of text-guided editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)