Title: FastDrag: Manipulate Anything in One Step
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Model;Drag-based Image Editing
Abstract: Drag-based image editing using generative models provides precise control over image contents, enabling users to manipulate anything in an image with a few clicks. However, prevailing methods typically adopt $n$-step iterations for latent semantic optimization to achieve drag-based image editing, which is time-consuming and limits practical applications. In this paper, we introduce a novel one-step drag-based image editing method, i.e., FastDrag, to accelerate the editing process. Central to our approach is a latent warpage function (LWF), which simulates the behavior of a stretched material to adjust the location of individual pixels within the latent space. This innovation achieves one-step latent semantic optimization and hence significantly promotes editing speeds. Meanwhile, null regions emerging after applying LWF are addressed by our proposed bilateral nearest neighbor interpolation (BNNI) strategy. This strategy interpolates these regions using similar features from neighboring areas, thus enhancing semantic integrity. Additionally, a consistency-preserving strategy is introduced to maintain the consistency between the edited and original images by adopting semantic information from the original image, saved as key and value pairs in self-attention module during diffusion inversion, to guide the diffusion sampling. Our FastDrag is validated on the DragBench dataset, demonstrating substantial improvements in processing time over existing methods, while achieving enhanced editing performance.

FastDrag: Manipulate Anything in One Step

Abstract

Drag-based image editing using generative models provides precise control over image contents, enabling users to manipulate anything in an image with a few clicks. However, prevailing methods typically adopt n -step iterations for latent semantic optimization to achieve drag-based image editing, which is time-consuming and limits practical applications. In this paper, we introduce a novel one-step dragbased image editing method, i.e. , FastDrag, to accelerate the editing process. Central to our approach is a latent warpage function (LWF), which simulates the behavior of a stretched material to adjust the location of individual pixels within the latent space. This innovation achieves one-step latent semantic optimization and hence significantly promotes editing speeds. Meanwhile, null regions emerging after applying LWF are addressed by our proposed bilateral nearest neighbor interpolation (BNNI) strategy. This strategy interpolates these regions using similar features from neighboring areas, thus enhancing semantic integrity. Additionally, a consistency-preserving strategy is introduced to maintain the consistency between the edited and original images by adopting semantic information from the original image, saved as key and value pairs in self-attention module during diffusion inversion, to guide the diffusion sampling. Our FastDrag is validated on the DragBench dataset, demonstrating substantial improvements in processing time over existing methods, while achieving enhanced editing performance. Project page: https://fastdrag-site.github.io/ .

1 Introduction

The drag editing paradigm [28, 15, 20] leverages the unique properties of generative models to implement a point-interaction mode of image editing, referred to as drag-based image editing. Compared with text-based image editing methods [18, 10, 3, 30], drag-based editing enables more precise spatial control over specific regions of the image while maintaining semantic logic coherence, drawing considerable attention from researchers.

However, existing methods typically involve n -step iterative semantic optimization in latent space to obtain optimized latent with desired semantic based on the user-provided drag instructions. They focus on optimizing a small region of the image at each step, requiring n small-scale and shortdistance adjustments to achieve overall latent optimization, leading to a significant amount of time. These optimization approaches can be categorized primarily into motion-based [23, 28, 15, 16, 32, 4] and gradient-based [20, 19] n -step iterative optimizations, as shown in Fig. 1(a). n -step iterations in motion-based methods are necessary to avoid abrupt changes in the latent space, preventing image

Original

semantic

Desired

semantic

n

-step Iterative

Optimization

or

Motion-

based

based

Original

semantic

Desired

semantic

One-step Optimization

obtained by LWF

j

v

j

v

(b) FastDrag with one-step warpage optimization.

Figure 1: (a) Existing methods usually require multiple iterations to transform an image from its original semantic to desired semantic; (b) Our method utilizes latent warpage function (LWF) to calculate the warpage vectors ( i.e. , v j ) to move each individual pixel on feature map and achieve semantic optimization in one step.

distortions and ensuring a stable optimization process. This is exemplified in DragDiffusion [28] and GoodDrag [32], which require 70 to 80 iterations of point tracking and motion supervision for optimization. In addition, gradient-based methods align sampling results with the drag instructions through gradient guidance [6]. In this way, they also require multiple steps due to the optimizer [13, 25] needing multiple iterations for non-convex optimization. For instance, DragonDiffusion [20] requires around 50 gradient steps to accomplish the latent optimization. Therefore, existing dragbased image editing methods often suffer from significant time consumption due to n -step iterations required for latent semantic optimization, thus limiting the practical applications.

To this end, we present a novel one-step drag-based image editing method based on diffusion, i.e. , FastDrag, which significantly accelerates editing speeds while maintaining the quality and precision of drag operations. Specifically, a novel one-step warpage optimization strategy is proposed to accelerate editing speeds, which can achieve the latent semantic optimization in a single step with an elaborately designed latent warpage function (LWF), instead of using motion or gradient-based n -step optimizations, as illustrated in Fig. 1(b). By simulating strain patterns in stretched materials, we treat drag instructions on the noisy latent as external forces stretching a material, and introduce a stretch factor in LWF, which enables the LWF to generate warpage vectors to adjust the position of individual pixels on the noisy latent with a simple latent relocation operation, thus achieving one-step optimization for drag-based editing. Meanwhile, a bilateral nearest neighbor interpolation (BNNI) strategy is proposed to enhance the semantic integrity of the edited content, by interpolating null values using similar features from their neighboring areas to address semantic losses caused by null regions emerging after latent relocation operation, thus enhancing the quality of the drag editing.

Additionally, a consistency-preserving strategy is introduced to maintain the consistency of the edited image, which adopts the original image information saved in diffusion inversion ( i.e. , key and value pairs of self-attention in the U-Net structure of diffusion model) to guide the diffusion sampling for desired image reconstruction, thus achieving precise editing effect. To further reduce time consumption for inversion and sampling, the latent consistency model (LCM) [17] is employed in the U-Net architecture of our diffusion-based FastDrag. Therefore, our FastDrag can significantly accelerate editing speeds while ensuring the quality of drag effects.

Experiments on DragBench demonstrate that the proposed FastDrag is the fastest drag-based editing method, which is nearly 700% faster than the fastest existing method ( i.e. , DiffEditor [19]), and 2800% faster than the typical baseline method ( i.e. , DragDiffusion [28]), with comparable editing performance. We also conduct rigorous ablation studies to validate the strategies used in FastDrag.

Contributions: 1) We propose a novel drag-based image editing approach based on diffusion i.e. , FastDrag, where a LWF strategy is proposed to achieve one-step semantic optimization, tremendously enhancing the editing efficiency. 2) We propose a novel interpolation method ( i.e. , BNNI), which effectively addresses the issue of null regions, thereby enhancing the semantic integrity of the edited content. 3) We introduce a consistency-preserving strategy to maintain the image consistency during editing process.

2 Related Work

2.1 Text-based Image Editing

Text-based image editing has seen significant advancements, allowing users to manipulate images through natural language instructions. DiffusionCLIP [12] adopts contrastive language-image pre-

training (CLIP) [24] for diffusion process fine-tuning to enhance the diffusion model, enabling high-quality zero-shot image editing. The study in [7] manipulates the cross-attention maps within the diffusion process and achieves text-based image editing. Imagic [11] further enhances these methods by optimizing text embeddings and using text-driven fine-tuning of the diffusion model, enabling complex semantic editing of images. InstructPix2Pix [1] leverages a pre-trained large language model combined with a text-to-image model to generate training data for a conditional diffusion model, allowing it to edit images directly based on textual instructions during forward propagation. Moreover, Null-text Inversion [18] enhances text-based image editing by optimizing the default null-text embeddings to achieve desired image editing. Although text-based image editing methods enable the manipulation of image content using natural language description, they often lack the precision and explicit control provided by drag-based image editing.

2.2 Drag-based Image Editing

Drag-based image editing achieves precise spatial control over specific regions of the image based on user-provided drag instructions. Existing drag-based image editing methods generally rely on n -step latent semantic optimization in latent space to achieve image editing. These methods fall into two main categories: motion-based [23, 28, 32, 4, 16, 15, 9] and gradient-based [20, 19] optimizations. For example, DragGAN [23] employs generative adversarial network (GAN) for drag-based image editing with iterative point tracking and motion supervision steps. However, the image quality of the methods using GAN for image generation is worse than diffusion models [5]. Therefore, a series of diffusion-based methods have been proposed for drag-based image editing. For instance, DragDiffusion [28] employs iterative point tracking and motion supervision for latent semantic optimization to achieve drag-based editing. Building on this foundation, GoodDrag [32], StableDrag [4], DragNoise [16], and FreeDrag [15] have made significant improvements to the motionbased methods. Without coincidence, by utilizing feature correspondences, DragonDiffusion [20] and its improved version DiffEditor [19] formulate an energy function that conforms to the desired editing results, thereby transforming the image editing task into a gradient-based process that enables dragbased editing. However, these methods inherently require n -step iterations for latent optimization, which significantly increases the time consumption. Although SDEDrag [22] does not require n -step iterative optimization, it is still time-consuming due to the stochastic differential equation (SDE) process for diffusion. In addition, while EasyDrag [9] offers user-friendship editing, its requirement for over 24GB of memory ( i.e. , a 3090 GPU) limits its broad applicability. To this end, based on latent diffusion model (LDM) [26], we propose a novel one-step optimization method that substantially accelerates the image editing speeds.

3 Proposed Method

FastDrag is based on LDM [26] to achieve drag-based image editing across four phases. The overall framework is given in Fig. 2, and the detailed description of strategies in FastDrag are presented as follows: (1) Initially, FastDrag is based on a traditional image editing framework including diffusion inversion and sampling processes, which will be elaborated in Sec. 3.1. (2) The core phase in Sec. 3.2 is a one-step warpage optimization, employing LWF and a latent relocation operation to simulate the behavior of stretched material, allowing for fast semantic optimization. (3) BNNI is then applied in Sec. 3.3 to enhance the semantic integrity of the edited content, by interpolating the null regions emerging after the one-step warpage optimization. (4) The consistency-preserving strategy is introduced in Sec. 3.4 to maintain the desired image consistency with original image, by utilizing the key and value of self-attention in inversion to guide the sampling.

3.1 Diffusion-based Image Editing

Similar to most existing drag editing methods [28, 32, 20], FastDrag is also built upon diffusion model ( i.e. , LDM), including diffusion inversion and diffusion sampling.

Diffusion Inversion [29] is about mapping a given image to its corresponding noisy latent representation in the model's latent space. We perform semantic optimization on the noisy latent z t ∈ R w × h × c , due to it still captures the main semantic features of the image but is perturbed by noise, making it suitable as a starting point for controlled modifications and sampling [28]. Here, w , h , c represent the width, height and channel of z t , respectively. This process for a latent variable at diffusion step t can

Drag operation

Encoder

Original Image

Decoder

Diffusion Inversion

Diffusion Sampling

Desired Image

Warpage

Vector

One-step Warpage

Optimization

Function

Image with

drag operation

: Set of points in mask region

: Set of drag instruction

4

?

3

4

5

Bilateral Nearest Neighbor

Interpolation

（

BNNI

）

:

: Optimized     with null point (

𝒩

)

Preserving

U-Net

t

z



0

z



)

,

(

K V

t

z

)

(

loc

loc

w

=



=



l

u r d

loc

loc

loc

w

ref

,

,

,

t

z

 

Low

High

𝒩 𝒩

𝒩

𝒩

𝒩

u

ref

l

len

0

I

0

I



P

P

Latent Relocation

:

j

th warpage vector in

t

z

t

z



: Noisy latent

: Optimized

t

z

t

z

𝓥

j

v

|

|

j

v

j

v

FWR

(

P

,    )

𝓥

0

z

𝓥

D

D

)

,

(

K V

: Key and value pairs of self-attention

t

z

 

)

,

,

(

j

f

LWF

P D

l

ref

be expressed as:

where z 0 = E ( I 0 ) denotes the initial latent of the original image I 0 from the encoder [14] E ( · ) . α t is the noise variance at diffusion step t , and ϵ t is the noise predicted by U-Net. Subsequently, we perform a one-step warpage optimization on z t in Sec. 3.2.

Diffusion Sampling reconstructs the image from the optimized noisy latent z ′ t by progressively denoising it to the desired latent z ′ 0 . This sampling process can be formulated as:

where ϵ is the Gaussian noise and σ denotes the noise level. By iterating the process from t to 1, z ′ 0 is reconstructed, and the desired image can be obtained by I ′ 0 = D ( z ′ 0 ) , with D ( · ) being the decoder [14].

3.2 One-step Warpage Optimization

Building upon the phases in Sec.3.1, we propose a one-step warpage optimization for fast drag-based image editing. The core idea involves simulating strain patterns in stretched materials, where drag instructions on the noisy latent are interpreted as external forces stretching the material. This enables us to adjust the position of individual pixels on the noisy latent, optimizing the semantic of noisy latent in one step, thus achieving extremely fast drag-based editing speeds. To this end, we design the LWF in Sec. 3.2.1 to obtain warpage vector, which is utilized by a straightforward latent relocation operation in Sec. 3.2.2 to adjust the position of individual pixels on the noisy latent.

3.2.1 Warpage Vector Calculation using LWF

In drag-based image editing, each drag instruction d i in a set of k drag instructions D = { d i | i = 1 , . . . , k ; k ∈ Z } can simultaneously influence a feature point p j on the mask region P = { p j | j = 1 , . . . , m ; m ∈ Z } provided by the user. As shown in Fig. 3, the mask region is represented by the brighter areas in the image, indicating the specific image area to be edited. To get a uniquely determined vector, i.e. , warpage vector v j to adjust the position of feature point p j (will be discussed in Sec.3.2.2), we propose a latent warpage function f LWF ( · ) to aggregate multiple component warpage vectors caused by different drag instructions, i.e. , v i ∗ j , with balanced weights to avoid

deviating from the desired drag effect. The function is given as follows:

where w i j is the normalization weight for component warpage vector v i ∗ j . Here, drag instruction d i is considered as a vector form handle point s i to target point e i . During dragging, we aim for the semantic changes around the handle point s i to be determined by the corresponding drag instruction d i , rather than other drag instructions far from the s i . Therefore, w i j is calculated as follows:

where s i is considered as the 'point of force' of d i , and the weight w i j is inversely proportional to the Euclidean distance from s i to p j .

It is worth noting that under an external force, the magnitude of component forces at each position within the material is inversely proportional to the distance from the force point, while the movement direction at each position typically aligns with the direction of the applied force [21]. Similarly, the component warpage vector v i ∗ j on each p j aligns with the direction of drag instruction d i , and magnitudes of v i ∗ j are inversely proportional to the distance from s i . Hence, v i ∗ j can be simplified as:

where λ i j is the stretch factor that denotes the proportion between v i ∗ j and d i .

To appropriately obtain the stretch factor λ i j and facilitate the calculation, we delve into the geometric representation of the component warpage vector v i ∗ j . As shown in Fig. 3, v i ∗ j can be depicted as the guidance vector from point p j to point p i ∗ j , where p i ∗ j is the expected new position of p j under the drag effect of d i . Recognizing that the content near to mask edge should remain unaltered, we define a reference circle O where every v i ∗ j will gradually reduce to 0 as p j approaches the circle. Consequently, since v i ∗ j and d i are parallel, magnitudes of v i ∗ j are inversely proportional to the distance from s i and v i ∗ j is reduced to 0 on circle O , the extended lines from s i p j and e i p i ∗ j will intersect at q i j on circle O . Hence, based

si

ei

O

*

i

j

p

i

j

q

si

ei

O

j

p

i

d

on the Eq. (5) and the geometric principle in Fig. 3, we calculate λ i j as follows:

Finally, we obtain the warpage vector v j using only d i and two factors as follows:

Note that, for the special application of drag-based editing, such as object moving as shown in Fig. 8, drag editing is degenerated to a mask region shifting operation, requiring the spatial semantics of the mask region to remain unchanged. In that case, we only process a single drag instruction, and all component drag effects will be set equal to the warpage vector, i.e. , v j = d 1 and D = { d 1 }.

3.2.2 Latent Relocation with Warpage Vector

Consequently, we utilize the warpage vector v j to adjust the position of feature point p j via a latent relocation operation F WR , achieving the semantic optimization of noisy latent for drag-based editing.

3

2

4

6

1

2

2

2

3

2

4

3

4

3

5

4

?

3

4

5

lenu=2

lend=2

lenr=1

lenl=3



=

=

l

u r d

loc

loc

loc

loc

len

1

len

1

w

,

,

,

/

/

ref

u

BNNI

:

ref

BNNI

WR

F

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

𝒩

t

z

t

z

 

t

z



:

𝒩

(null point)



=



l

u r d

loc

loc

loc

w

ref

,

,

,

KV

Q

KV

Q

KV

Q

KV

Q

: U-Net in inversion process

: Self-Attention module

KV

Q

KV

/ Q

: U-Net in sampling process

KV

Q

KV

Q

KV

Q

KV

Q

Establishing a Cartesian coordinate system on the latent space, let ( x p j , y p j ) denote the position of point p j ∈ P within this coordinate system. The new location of the point p ∗ j after applying the vector v j = ( v x j , v y j ) can be written as:

Then the new coordinates set C of all feature points in P can be written as:

where V = { v j | j = 1 , · · · , m, m ∈ Z } . If ( x p j , y p j ) has already been a new position for a feature point, it no longer serves as a new position for any other points. Consequently, by assigning corresponding values to these new positions, the optimized noisy latent z ′′ t can be obtained as shown in the following equation:

In essence, the latent relocation operation optimizes semantics efficiently by utilizing the LWFgenerated warpage vector, eliminating the need for iterative optimization.

However, as certain positions in the noisy latent may not be occupied by other feature points, z ′′ t obtained from one-step warpage optimization may contain regions with null values as shown in Fig. 4, leading to semantic losses that can adversely impact the drag result. We address this issue in Sec. 3.3.

3.3 Bilateral Nearest Neighbor Interpolation

To enhance the semantic integrity, BNNI interpolates points in null region using similar features from their neighboring areas in horizontal and vertical directions, thus ensuring the semantic integrity and enhancing the quality of drag editing. Let N be a point with coordinate ( x N , y N ) in null regions, we identify the nearest points of N containing value in four directions: up, right, down, and left, as illustrated in Fig. 4, which are used as reference points for interpolation. Then, the interpolated value for null point N can be calculated as:

where ref loc denotes the value of reference point, and loc indicates the direction, with u , r , d and l representing up, right, down and left, respectively. w loc is the interpolation weight for each reference point, which is calculated based on its distance to N , as follows:

where len loc represents the distance between the reference point and N . Such that we can obtain the optimized noisy latent z ′ t with complete semantic information by using BNNI to exploit similar semantic information from surrounding areas, further enhancing the quality of the drag editing.

FastDrag (Ours)

DragDiffusion

FreeDrag

DragNoise

User Edit

3.4 Consistency-Preserving Strategy

Following [20, 2, 28], we introduce a consistency-preserving strategy to maintain the consistency between the edited image and the original image by adopting the semantic information of the original image ( i.e. , key and value pairs) saved in self-attention module during diffusion inversion to guide the diffusion sampling, as illustrated in Fig. 5. Specifically, during the diffusion sampling, the calculation of self-attention AttentionSa within the upsampling process of the U-Net is as follows:

where query Q Sa is still used from diffusion sampling but key K In and value V In are correspondingly from diffusion inversion. Thus, the consistency-preserving strategy maintains the overall content consistency between the desired image and original image, ensuring the effect of drag-based editing.

4 Experiments

4.1 Qualitative Evaluation

We conduct experiments to demonstrate the drag effects of our FastDrag method, comparing it against state-of-the-art techniques such as DragDiffusion [28], FreeDrag [15], and DragNoise [16]. The qualitative comparison results are presented in Fig. 6. Notably, FastDrag maintains effective drag performance and high image quality even in images with complex textures, where n -step iterative methods typically falter. For instance, as shown in the first row of Fig. 6, FastDrag successfully rotates the face of an animal while preserving intricate fur textures and ensuring strong structural integrity. In contrast, methods like DragDiffusion and DragNoise fail to rotate the animal's face, and FreeDrag disrupts the facial structure.

In the stretching task, FastDrag outperforms all other methods, as shown in the second row of Fig. 6, where the goal is to move a sleeve to a higher position. The results show that other methods lack robustness to slight deviations in user dragging, where the drag point is slightly off the sleeve.

4

6

8

10 12 14

20

30

Number of Inversion Steps t

26

28

30

32

34

36

38

Mean Distance

Mean Distance

0.80

0.81

0.82

0.83

0.84

0.85

0.86

0.87

Image Fidelity (1-LPIPS)

Image Fidelity (1-LPIPS)

Despite this, FastDrag accurately moves the sleeve to the desired height, understanding the underlying semantic intent of dragging the sleeve.

Additionally, we perform multi-point dragging experiments, illustrated in the fourth row of Fig. 6. Both DragDiffusion and DragNoise fail to stretch the back of the sofa, while FreeDrag incorrectly stretches unintended parts of the sofa. Through the LWF introduced in Sec. 3.2.1, FastDrag can manipulate all dragged points to their target locations while preserving the content in unmasked regions. More results of FastDrag are illustrated in supplementary Sec. E.

4.2 Quantitative Comparison

To better demonstrate the superiority of FastDrag, we conduct quantitative comparison using DragBench dataset [28], which consists of 205 different types of images with 349 pairs of handle and target points. Here, mean distance (MD) [23] and image fidelity (IF) [11] are employed as performance metrics, where MD evaluates the precision of drag editing, and IF measures the consistency between the generated and original images by averaging the learned perceptual image patch similarity (LPIPS) [31]. Specifically, 1-LPIPS is employed as the IF metric in our experiment to facilitate comparison. In addition, we compare the average time required per point to demonstrate the time efficiency of our proposed FastDrag. The results are given in Table 1.

Apart from [28],[16],[15], two other state-of-the-art methods, i.e. , GoodDrag [32] and DiffEditor [19], are also adopted for comparison, with DiffEditor being the current fastest drag-based editing method. Due to well-designed one-step warpage optimization and consistency-preserving strategy, our FastDrag does not require LoRA training preparation, resulting in significantly reduced time consumption ( i.e. , 3.12 seconds), which is nearly 700% faster than DiffEditor ( i.e. , 21.68 seconds), and 2800% faster than the typical baseline DragDiffusion ( i.e. , 1 min and 21.54 seconds). Moreover, even using standard U-Net without LCM, our method is still much faster than DiffEditor and far outperforms all other state-of-the-art methods. It is particularly noteworthy that, even with an A100 GPU, DiffEditor still requires 13.88 seconds according to [19], whereas FastDrag only requires 3.12 seconds on an RTX 3090.

In addition, our FastDrag also achieves competitive quantitative evaluation metrics ( i.e. , IF and MD) comparable to the state-of-the-art methods, and even better drag editing quality, as illustrated in Fig. 6. These results demonstrate the effectiveness and superiority of our method.

4.3 Ablation Study

Inversion Step: To determine the number of inversion steps in diffusion inversion with LCMequipped U-Net, we conduct an ablation experiment with number of inversion steps set as t = 4, 6, 8, 10, 12, 14, 20, and 30, where IF and MD are used to evaluate the balance between the consistency with original image and the desired drag effects. The results are given in Fig. 8 and Fig. 7, where we can see that when t < 6 , the generated images lack sufficient detail to accurately reconstruct the original images. Conversely, when t > 6 , it can successfully recover complex details such as intricate fur textures and dense stone while maintaining high image quality. However, when t > 14 , some image details lost, which negatively impacts the effectiveness of the drag effect. By comprehensive

4 steps

6 steps

8 steps

10 steps

12 steps

14 steps

20 steps

30 steps

Original image

Editing

Figure 8: Ablation study on number of inversion steps in terms of drag effect.

BNNI

Original value

0 interpolation

Random interpolation

User edit

C

evaluation of both the drag effect and the similarity to the original images, we select 10 as the number of inversion steps for our method with LCM to balance the drag effect.

BNNI: To demonstrate the effectiveness of BNNI, we compare it with several interpolation methods on null point N , including maintaining the original value of this position, interpolation by zero-value, and interpolation by random noise, denoted as 'original value', '0 interpolation', and 'random interpolation', respectively. The results are given in Fig. 9, where we can see that, by effectively utilizing surrounding feature values to interpolate null points, BNNI can address semantic losses, and enhance the quality of the drag editing.

Consistency-Preserving: We also conduct an experiment to validate the effectiveness of the consistency-preserving strategy in maintaining image consistency. The results are illustrated in Fig. 10, where 'w/ CP' and 'w/o CP' denote our FastDrag with and without using consistencypreserving strategy, respectively. It is obviously that our method with consistency-preserving strategy can effectively preserve image consistency, resulting in better drag editing effect.

User edit

w/o CP

w/ CP

w/o CP

w/ CP

User edit

DragDiffusion

FreeDrag

DragNoise

FastDrag (Ours)

User edit

Original Image

FastDrag

User edit

Original Image

FastDrag

User edit

Original Image

Failed

Succeed

(a) Desired editing effect: Thin the hair

while keeping the face size

(b) Desired editing effect:

Lengthen the beak

5 Limitations

Despite FastDrag's impressive editing speed compared to SOTA methods, it shares some common limitations. 1) Overly Smooth and Finer Details Loss: Similar to other diffusion-based methods [3, 28], FastDrag occasionally loses fine textures from the original images, as shown in Fig. 6, row 4. Despite this, FastDrag outperforms other methods in speed and overall performance. 2) Extremely Long-distance Drag Editing: In such case, object details may be lost due to the lower-dimensional latent space, in which significant changes in detail (i.e., long-drag editing) can disrupt the semantics, making it harder to preserve all details. Nevertheless, FastDrag handles long-distance editing better than other SOTA methods, as illustrated in Fig. 11, where our method successfully achieves longdistance drag editing that others fail to achieve. 3) Highly Relying on Precise Drag Instruction: Achieving optimal results depends heavily on clear drag instructions. As with other SOTA methods, precise input, such as excluding irrelevant areas from the mask (e.g., the face in Fig. 12, row 2) or correctly placing the handle point (e.g., beak in Fig. 12), is essential for better performance.

6 Conclusion

This paper has presented a novel drag-based image editing method, i.e. , FastDrag, which achieved faster image editing speeds than other existing methods. By proposing one-step warpage optimization and BNNI strategy, our approach achieves high-quality image editing according to the drag instructions in a very short period of time. Additionally, through the consistency-preserving strategy, it ensures the consistency of the generated image with the original image. Moving forward, we plan to continue refining and expanding our approach to further enhance its capabilities and applications.

7 Acknowledgments

This work was partly supported by Beijing Nova Program (20230484261).

References

Tim Brooks, Aleksander Holynski, and Alexei A. Efros. InstructPix2Pix: Learning to follow image editing instructions. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2023.

Ming Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. MasaCtrl: Tuningfree mutual self-attention control for consistent image synthesis and editing. Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV) , 2023.

Hansam Cho, Jonghyun Lee, Seoung Bum Kim, Tae-Hyun Oh, and Yonghyun Jeong. Noise map guidance: Inversion with spatial context for real image editing. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2024.

Yutao Cui, Xiaotong Zhao, Guozhen Zhang, Shengming Cao, Kai Ma, and Limin Wang. StableDrag: Stable dragging for point-based image editing. arXiv preprint arXiv:2403.04437 , 2024.

Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , 2021.

Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , 2023.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Promptto-prompt image editing with cross attention control. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2023.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS) , 2021.

Xingzhong Hou, Boxiao Liu, Yi Zhang, Jihao Liu, Yu Liu, and Haihang You. Easydrag: Efficient pointbased manipulation on diffusion models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , pages 8404-8413, 2024.

Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Direct inversion: Boosting diffusionbased editing with 3 lines of code. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2024.

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2023.

Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. DiffusionCLIP: Text-guided diffusion models for robust image manipulation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR , abs/1312.6114, 2013.

Pengyang Ling, Lin Chen, Pan Zhang, Huaian Chen, Yi Jin, and Jinjin Zheng. FreeDrag: Feature dragging for reliable point-based image editing. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2024.

Haofeng Liu, Chenshu Xu, Yifei Yang, Lihua Zeng, and Shengfeng He. Drag your noise: Interactive point-based editing via diffusion semantic propagation. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2024.

Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378 , 2023.

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2023.

Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. DiffEditor: Boosting accuracy and flexibility on diffusion-based image editing. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2024.

Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. DragonDiffusion: Enabling drag-style manipulation on diffusion models. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2024.

D. Naylor. Theoretical elasticity, by A. E. Green and W. Zerna (Second edition). Clarendon Press, Oxford, 1968. xv + 457 pages. Can. Math. Bull , 12:537-538, 1969.

Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, and Chongxuan Li. The blessing of randomness: SDE beats ODE in general diffusion-based image editing. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2024.

Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your GAN: Interactive point-based manipulation on the generative image manifold. In Proc. ACM Conf. SIGGRAPH , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Machine Learning (ICML) , 2021.

Herbert E. Robbins. A stochastic approximation method. Ann. Math. Stat. , 22:400-407, 1951.

Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2022.

Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and Song Bai. DragDiffusion: Harnessing diffusion models for interactive point-based image editing. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2024.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proc. Int. Conf. Learn. Represent. (ICLR) , 2021.

Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with natural language. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2024.

Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2018.

Zewei Zhang, Huan Liu, Jun Chen, and Xiangyu Xu. GoodDrag: Towards good practices for drag editing with diffusion models. arXiv preprint arXiv:2404.07206 , 2024.

A Supplementary Experiments

Weconduct supplementary quantitative experiments on BNNI and consistency-preserving strategies to further validate their effectiveness. The quantitative metrics used are consistent with those described in Sec. 4.1.

BNNI: Following the setup in Sec.4.3, we compare it with several interpolation methods on null point N , including maintaining the original value of this position, interpolation by zero-value, and interpolation by random noise, denoted as 'origin', '0-inter', and 'random-inter', respectively. As illustrated in Fig. 13, FastDrag with BNNI achieves the best MD levels compared to other interpolation methods, while its IF is second only to 'origin'. However, 'origin' can lead to negative drag effects, as shown in Fig. 9. Therefore, by effectively utilizing surrounding feature values to interpolate null points, BNNI can address semantic losses and enhance the quality of drag editing.

Consistency-Preserving: We also conduct experiments to assess the impact of initiating the consistency-preserving strategy at different sampling steps. The results, as shown in Fig. 14, indicate that as the starting step increases ( i.e. , the frequency of key and value replacements decreases), the IF decreases, leading to poorer image consistency. Meanwhile, the MD initially decreases and then increases as the starting step increases. It is evident that consistency-preserving strategy can effectively maintain the consistency between the generated images and the original images.

origin

0-inter

random-inter

BNNI

Interpolation Strategy

32.5

33.0

33.5

34.0

34.5

35.0

35.5

Mean Distance

Mean Distance

0.8525

0.8550

0.8575

0.8600

0.8625

0.8650

0.8675

0.8700

Image Fidelity (1-LPIPS)

Image Fidelity (1-LPIPS)

B Single-step Diffusion

When integrating DragDiffusion with a singlestep diffusion model, the editing time is still much longer than that of FastDrag. For DragDiffusion and FastDrag under diffusion steps of 1, 20, and 50, we calculate the time required for inversion, sampling, and latent optimization respectively. The results provided in Fig. 15 show that even with a single diffusion step (i.e., diffusion step set as 1), DragDiffusion still requires significantly more time (20.7 seconds) compared to FastDrag (2.88 seconds).

In addition, as observed in Fig. 15, DragDiffusion spends significantly more time on latent optimization compared to diffusion inversion and sampling. Therefore, reducing the time spent on latent optimization is crucial for minimizing overall editing time, which is precisely what FastDrag accomplishes.

25

20

Time(s)

10

5

0

FastDrag(20)

FastDrag(50)

DragDiffusion(1)

DragDiffusion(20)

DragDiffusion(50)

Methods with different diffusion steps

■inversion

■sampling

optimization

3

4

5

6

7

8

Start step of Consistency-Preserving

32

33

34

35

36

Mean Distance

Mean Distance

0.80

0.81

0.82

0.83

0.84

0.85

Image Fidelity (1-LPIPS)

Image Fidelity (1-LPIPS)

C Statistical Rigor

To further validate the superiority of our work and to achieve statistical rigor, we conducted an additional experiment by repeating our experiment 10 times under the same experimental settings. We observed that the variances of the performance metrics obtained from 10 realizations of our FastDrag are MD (0.000404), 1-LPIPS (9.44E-11), and Time (0.018), all of which fall within a reasonable range. These statistical results further demonstrate the effectiveness and stability of our method for drag editing.

D Implementation Details

We utilize a pretrained LDM ( i.e. , Stable Diffusion 1.5 [27]) as our diffusion model, where the U-Net structure is adapted with LCM-distilled weights from Stable Diffusion 1.5. It is worth emphasizing that the U-Net structure used in our model is widely used in image generation methods [28, 15, 16, 32, 4]. Unless otherwise specified, the default setting for inversion and sampling step is 10. Following DragDiffusion [28], classifier-free guidance (CFG) [8] is not applied in diffusion model, and we optimize the diffusion latent at the 7th step. All other configurations follow that used in DragDiffusion. Our experiments are conducted on an RTX 3090 GPU with 24G memory.

For the special application of drag-based editing, i.e. , object moving, as shown in Fig. 8, significant null region may be left at the original position of the object due to long-distance relocation, posing challenges for BNNI. To ensure semantic integrity in the image and facilitate user interaction, we adopted two straightforward strategies. For first strategy, specifically, we introduce a parameter r , set as 2, centered at the unique target point e 1 , defining a rectangular area with dimensions 2 r . We then extract the noised-latent representation within this rectangular area and fill it into the mask, effectively restoring the semantics at the original position of the object. For second strategy, we just maintain the original semantic of original position to avoid null region. Under second strategy, object moving will produce the effect of object replication, as shown in the third row of Fig. 16.

E More Results

We apply FastDrag to drag-based image editing in various scenarios, including face rotation, object movement, object stretching, object shrinking and so on. The experimental results, as shown in Fig. 16, demonstrate that FastDrag achieves excellent drag-based effects across multiple scenarios.

F Societal Impacts

FastDrag has the potential to bring about several positive societal impacts. Firstly, they offer intuitive and efficient image editing tools, catering to artists, designers, and creators, thereby fostering creativity and innovation. Secondly, their user-friendly nature simplifies the image editing process, increasing accessibility and participation among a wider audience. In addition, our FastDrag improves efficiency and saves the user's time and effort, thus increasing productivity. Moreover, the innovative and flexible nature of these methods opens up possibilities for various applications, spanning art creation, design, education, and training.

However, along with these positive aspects, FastDrag can also have certain negative societal implications. They may be exploited by unethical individuals or organizations to propagate misinformation and fake imagery, potentially contributing to the spread of false news and undermining societal trust. Furthermore, widespread use of image editing tools may encroach upon individual privacy rights, particularly when unauthorized information or imagery is manipulated. Moreover, inappropriate or irresponsible image editing practices could lead to social injustices and imbalances, such as distorting facts or misleading the public, thereby influencing public opinion and policies negatively.

Original image

User edit

Result

Original image

User edit

Result

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly outline the contributions of our approach in both the abstract and the introduction in Sec.1 of our paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the our method in Section 5

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not involve theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our experimental results are reproducible. We provide a detailed description of our method's workflow in Sec.3 and implementation details in Sec.D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include the implementation code for our method in the supplementary materials submitted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The main experimental details of our method are discussed in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our experiments, conducted on the DragBench dataset, yield average metrics that are statistically significant and provide a reliable basis for comparing our method with others. And we discuss experimental statistical significance in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided detailed information about the computer resources used for all our experiments in Appendix D

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have thoroughly reviewed the NeurIPS Code of Ethics and can confirm that our research conducted in the paper adheres to the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The potential positive societal impacts and negative societal impacts of the our work is discussed in Appendix F

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not introduce any new datasets or pretrained models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The models and baselines utilized in our method are detailed in Appendix D of the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not introduce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: During our research, we do not employ crowdsourcing or conduct studies involving human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: In our research process, human subjects were not required to assist with the study.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method that enables fast drag-based image editing using diffusion models. The proposed method uses a latent warpage function to obtain the dragged latent representation. The additional nearest neighbor interpolation and content-preserving strategy further improve the result.\n\nSTRENGTHS:\nThe strengths of this paper are:\n- The proposed method seems straightforward, and the results are convincing.\n- A large speed performance gain compared to the previous method.\n- The evaluations support the paper’s claim.\n\nWEAKNESSES:\nThe weaknesses of this paper are:\n- I think the discussion of limitation is too shallow. I recommend providing more failing cases and discussing the potential directions to enhance the results.\n\nQUESTIONS:\n- I wonder why the interpolation is called “Bilateral” nearest neighbor interpolation. For me, “Bilateral” term should be used for a scheme that considers the value as well. And the proposed interpolation scheme is just find the nearest non-null value. This is not critical but I think it is misleading.\n- It is unclear to me what the brighter region around the drag arrow means. Is that a mask? Are those region/mask included and used in drag instructions?\n- I am confused about Figure 8. What is the desired effect and result of the drag edit? Semantically, the user is moving “the dog’s” hand, not “the cat’s”. However, the result seem to generate cat’s hand for all different steps. Is this some limitation of all drag-based method? If yes, I think it is worth discussing.\n- I wonder if there are metrics or evaluation processes that can evaluate the content preserving quantitatively. Regarding image editing, I think this is a very important factor and the current paper only provides a couple of images as evidence.\n\nLIMITATIONS:\nAs I mentioned above, I think the limitation discussion is not enough (I also raised another potential limitation). Moreover, I strongly recommend including the limitation discussion in the main paper, not supplemental material.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces \"FastDrag,\" a novel one-step drag-based image editing method that significantly accelerates the editing process compared to existing n-step iterative approaches. The core of FastDrag is the Latent Warpage Function (LWF), which simulates the behavior of stretched material to adjust pixel locations within the latent space, enabling single-step latent semantic optimization. This results in substantially faster editing speeds while maintaining quality and precision.\n\nThe paper's contributions include:\n1. The proposal of FastDrag, a diffusion-based image editing approach that employs LWF for one-step semantic optimization, greatly enhancing editing efficiency.\n2. The introduction of Bilateral Nearest Neighbor Interpolation (BNNI), a new interpolation method that addresses null regions and enhances the semantic integrity of edited content.\n3. The incorporation of a consistency-preserving strategy that maintains image consistency during the editing process by using original image information to guide diffusion sampling.\n\nExperiments on the DragBench dataset demonstrate FastDrag's superiority in processing time, being nearly 700% faster than the fastest existing method and 2800% faster than the typical baseline method, with comparable editing performance. The paper also includes rigorous ablation studies to validate the strategies used in FastDrag. The authors aim to continue refining and expanding their approach for enhanced capabilities and applications in the future.\n\nSTRENGTHS:\n**Originality:** The paper introduces \"FastDrag,\" a novel one-step drag-based image editing method that significantly accelerates the editing process compared to conventional n-step iterative methods. By proposing a Latent Warpage Function (LWF), the authors innovate by simulating the behavior of a stretched material to instantly adjust pixel locations within the latent space, marking a departure from traditional iterative latent optimization. Additionally, the Bilateral Nearest Neighbor Interpolation (BNNI) strategy for handling null regions and the consistency-preserving mechanism that utilizes self-attention key-value pairs add further novelty to the approach. The integration of these components into a diffusion model framework for image editing showcases a creative combination of existing techniques to solve a known problem more efficiently.\n\n**Quality:** The quality of the work is demonstrated through rigorous experimentation on the DragBench dataset, which validates FastDrag's efficiency and performance. The paper shows that FastDrag is not only faster but also maintains editing quality, being nearly 700% faster than the fastest existing method and 2800% faster than typical baseline methods. A series of ablation studies further reinforces the effectiveness of the individual components of the proposed method. The computational efficiency and scalability implications are considered, even though specific computational requirements are not fully detailed in the abstracted content. \n\n**Clarity:** The paper is structured coherently, presenting the FastDrag method systematically across its phases: diffusion-based editing, one-step warpage optimization, BNNI for semantic integrity, and the consistency-preserving strategy. The abstract and introduction effectively outline the contributions, setting clear expectations for the paper's content. Theoretical assumptions and limitations are openly discussed, complying with the NeurIPS Code of Ethics. The use of visual aids like Fig. 2 likely contributes to illustrating the method's workflow, although the figure itself is not accessible in the abstracted content.\n\n**Significance:** The significance of FastDrag lies in its potential to democratize and streamline image editing tasks, enhancing productivity in various domains such as art, design, education, and training. By offering an intuitive and efficient tool, it empowers a broader range of users to engage in creative image manipulation without requiring extensive technical skills. The societal impact discussion highlights both the positive outcomes of fostering creativity and the potential negative implications like misuse for disinformation or privacy breaches, showing a balanced perspective. The authors' consideration of mitigation strategies underscores a proactive approach to responsible innovation.\n\nOverall, FastDrag represents a marked advancement in image editing technology, combining originality in method design, high-quality empirical validation, clarity in presentation, and significant implications for both the research community and practical applications. Its contribution to reducing latency in image editing workflows can have far-reaching effects on digital content creation workflows and user experiences.\n\nWEAKNESSES:\nThe paper titled \"FastDrag: Manipulate Anything in One Step\" presents a novel one-step drag-based image editing method that aims to accelerate the editing process compared to existing n-step iterative optimization methods. While the paper makes significant contributions to the field of image editing, there are several areas where the work could be improved:\n\n1. **Theoretical Depth**: The paper does not appear to include theoretical results or proofs to support the claims made. While it is stated that the paper does not involve theoretical results, providing some theoretical grounding could strengthen the contribution, such as explaining the mathematical properties of the proposed Latent Warpage Function (LWF) or the bilateral nearest neighbor interpolation (BNNI) strategy.\n\n2. **Experimental Design**: While the paper includes qualitative and quantitative evaluations, there could be a more extensive range of experiments to further validate the robustness of the method. For instance, testing the method on a broader variety of image types and editing tasks could provide a more comprehensive understanding of its capabilities and limitations.\n\n3. **Statistical Rigor**: The paper does not report error bars or other statistical measures that would provide insight into the variability and reliability of the results. Including such measures would enhance the credibility of the experimental findings.\n\n4. **Comparison with State-of-the-Art**: The paper compares its method with existing techniques but could benefit from a more detailed analysis of where FastDrag stands in terms of the trade-offs between speed and quality. A deeper dive into how the method's performance compares across different types of image content and editing tasks would be valuable.\n\n5. **User Study**: The paper lacks a user study that could provide insights into the usability and user experience of the FastDrag method. Including feedback from potential end-users, such as artists or designers, could highlight practical aspects that may not be evident from quantitative metrics alone.\n\n6. **Limitations Discussion**: While the paper does discuss some limitations, it could be more explicit about the scenarios where the method may not perform as expected. Providing more concrete examples and discussing potential workarounds or future work to address these issues would be beneficial.\n\n7. **Reproducibility**: The paper claims that the results are reproducible and that the code is included in the supplementary materials. However, it would be helpful to provide more detailed instructions on how to set up the environment and run the experiments, ensuring that other researchers can easily replicate the results.\n\n8. **Ethical Considerations**: The paper touches on societal impacts but could expand on the ethical considerations of using such technology, especially regarding the potential for misuse in spreading misinformation or altering reality.\n\n9. **Technical Details**: The paper could benefit from a more thorough explanation of the technical details, such as the specific design choices in the U-Net architecture and how the latent consistency model (LCM) is integrated into the FastDrag framework.\n\n10. **Long-Term Performance**: The paper could address how the method performs over time, especially considering the potential for the model to degrade in quality as more edits are made to the same image.\n\nBy addressing these points, the authors could strengthen the paper's contributions and provide a more comprehensive understanding of the FastDrag method's capabilities and potential areas for future research.\n\nQUESTIONS:\n**Questions for the Authors:**\n\n1. **Latent Warpage Function (LWF) Details:** Could you provide more technical details about the design and implementation of the Latent Warpage Function? Specifically, how is the stretching behavior modeled, and what criteria determine the degree and direction of pixel adjustments within the latent space?\n\n2. **Bilateral Nearest Neighbor Interpolation (BNNI) Robustness:** How does the BNNI strategy cope with complex scenes or objects with varied textures? Are there any specific cases where BNNI might struggle, and how do these instances affect the final image quality?\n\n3. **Consistency-Preserving Strategy and Diffusion Sampling:** Can you elaborate on the process of adopting semantic information from the original image during diffusion inversion, and how exactly these key-value pairs guide the sampling process to maintain consistency? Are there quantitative measures to evaluate the consistency preservation?\n\n4. **Scalability with Image Complexity and Dataset Size:** Your work demonstrates impressive speed improvements, but how does FastDrag's performance scale with the complexity of the image being edited or the size of the underlying dataset? Are there any benchmarks or theoretical analyses to predict computational demands for larger datasets or more intricate images?\n\n5. **Addressing Negative Societal Implications:** Given the potential misuse of FastDrag for generating disinformation or breaching privacy, have you explored or considered integrating any technical safeguards directly into the tool to mitigate these risks? For instance, adding watermarks, traceability features, or built-in detection mechanisms for manipulated content.\n\n6. **Limitations and Future Work:** In Appendix D, you discuss the limitations of your method. Could you elaborate on the most critical limitations that future research should address to improve FastDrag, and what are your thoughts on potential directions for overcoming these limitations?\n\n7. **Error Analysis and Sensitivity to Initialization:** How sensitive is FastDrag to the initial state of the latent space? Have you conducted experiments to analyze the variability in output quality based on different starting points, and if so, could you share insights on the stability of the method?\n\n**Suggestions for Improvement:**\n\n1. **Quantitative Evaluation of Semantic Integrity:** Consider including a quantitative evaluation of the semantic integrity of edited images, perhaps through metrics that assess feature similarity or structure preservation, to complement qualitative assessments.\n\n2. **User Study or Survey:** Conduct a user study or survey to gather feedback from artists, designers, and other potential users to understand the real-world usability and satisfaction with FastDrag. This could validate the claimed benefits of intuitiveness and efficiency.\n\n3. **Expand on Mitigation Strategies:** Further elaborate on potential mitigation strategies for the negative societal impacts mentioned. Discuss how these strategies could be practically implemented, and if possible, provide examples of successful implementations in similar contexts.\n\n4. **Code Availability and Reproducibility:** Since the computational efficiency and resource requirements are crucial aspects of FastDrag, consider making the code publicly available with clear documentation to facilitate reproduction of your results and encourage further development by the community.\n\n5. **Additional Experimentation on Diverse Datasets:** Extend the evaluation to include a variety of datasets covering different image types and complexities to strengthen the generalizability of FastDrag's performance claims.\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work in the \"Limitations\" section of the paper, which is in line with the NeurIPS Paper Checklist guidelines. They are encouraged to explicitly state any assumptions made and discuss how these assumptions could be violated in practical scenarios, as well as the potential implications of such violations. For instance, if the FastDrag method is highly dependent on certain image characteristics, such as resolution or lighting, the authors should clarify these dependencies and discuss the possible decrease in performance under varying conditions.\n\nRegarding potential negative societal impacts, the authors should discuss if there are any direct paths from their research to unintended or malicious uses, as outlined in the guidelines. Given that FastDrag is an image editing tool that could potentially be misused for creating disinformation or deepfakes, it is essential for the authors to acknowledge this possibility and outline strategies to mitigate such risks. This could include discussing ethical release strategies for the model, such as gated access, user guidelines, or incorporating detection mechanisms to identify manipulated content.\n\nIf the authors have not done so already, they should also reflect on the broader ethical implications of their work, such as fairness considerations (ensuring the technology doesn't unfairly impact specific groups), privacy concerns (ensuring personal data is protected), and security considerations (preventing unauthorized use). The authors should also consider including a discussion on how the technology could be used as intended but still give incorrect results, or how it might be intentionally or unintentionally misused, along with possible mitigations.\n\nIn summary, if the authors have not adequately addressed these aspects, they should revise their paper to include a clear limitations section that outlines any assumptions, constraints, or areas where the method may fall short. Additionally, they should discuss potential negative societal impacts and propose strategies to mitigate these risks. The NeurIPS guidelines emphasize that authors should be transparent about limitations and negative impacts, and doing so will not negatively affect their review but instead demonstrates responsible research conduct.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-22"}

comments: SUMMARY:
This paper introduces a new one-step drag-based image editing method that significantly accelerates the editing process using a LWF function. It also employs a BNNI strategy to handle null regions and a consistency-preserving strategy to maintain the integrity of the edited image. Experimental results demonstrate FastDrag’s fast speed and performance compared to existing methods.

STRENGTHS:
1. FastDrag is easy for editing and has a fast editing speed.
2. BNNI strategy addresses the issue of null regions, maintaining semantic integrity and quality.
3. It also provides spatial control over specific regions of the image, enabling detailed drag editing.

WEAKNESSES:
1. If the drag distance is long, will the BNNI still success to maintain high semantic quality, how about the editing speed and complexity for long-distance dragging and latent relocation?
2. Threre should also be some failed examples to better illustrate the proposed method.
3. I am curious whether using a better base model can achieve better editing results, and whether there is a trade-off between editing time and editing performance.
4. More recent works [1] should be included for comparison

[1] EasyDrag: Efficient Point-based Manipulation on Diffusion Models, CVPR 2024

QUESTIONS:
See the strengths and weaknesses

LIMITATIONS:
See the strengths and weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a drag-based image editing method that uses the Latent Warpage Function to optimize pixel adjustments in a single step, which is an improvement over previous iterative methods. This approach simulates a stretched material in the latent space to allow for fast and accurate pixel adjustments. It also combines bilateral nearest-neighbor interpolation to handle null regions and a consistency-preserving strategy, involving key and value pairs in self-attention during the inversion process, to maintain semantic consistency and coherence.\n\nSTRENGTHS:\nThe proposed method reduces computational time by performing one-step optimization, which makes the process faster and more efficient. Additionally, it includes clever tricks using bi-linear nearest neighbor interpolation to fix empty areas. Preserving keys and values during inversion also appears to help maintain semantic coherence.\n\nWEAKNESSES:\nThe paper needs to include more comprehensive evaluation metrics beyond visual inspection. The chosen metrics are unreliable, and the evaluation seems to be on a very small scale. Moreover, there are missing standard errors, and it is unclear if the results are cherry-picked or randomly chosen as the numbers on the proposed metrics look very much alike. To improve the overall evaluation of editing accuracy and quality, using interest points and key points like SIFT, SuperPoint, or DUST3R to verify if features at desired locations match the correct points in the edited image would be beneficial. Furthermore, there is a need to explore the effect of the edits on \"other\" parts of the image to understand potential unintended alterations due to the stochasticity in diffusion models. The edited images appear overly smooth and lose finer details, which may impact the overall realism.\n\nQUESTIONS:\n1) Does the reported computational time include the inversion process?\n2) What happens if you integrate DragDiffusion with a single-step diffusion model? How does it compare with Fastdrag?\n3) Why does iterative drag diffusion suffer compared to the one-step approach?\n4) How does FastDrag affect other parts of the image during edits?\n5) Can you perform multi-point drag editing with FastDrag?\n\nLIMITATIONS:\nYes in the appendix", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}