DragEntity:Trajectory Guided Video Generation using Entity and Positional Relationships

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 15:56:01

Abstract: In recent years, diffusion models have achieved tremendous success in the field of video generation, with controllable video generation receiving significant attention. However, existing control methods still face two limitations: Firstly, control conditions (such as depth maps, 3D Mesh) are difficult for ordinary users to obtain directly. Secondly, it’s challenging to drive multiple objects through complex motions with multiple trajectories simultaneously. In this paper, we introduce DragEntity, a video generation model that utilizes entity representation for controlling the motion of multiple objects. In comparison to previous methods, MotionCtrl offers two main advantages: 1) Trajectory-based methods are more user-friendly for interaction. Users only need to draw trajectories during the interaction to generate videos. 2) We use entity representation to represent any object in the image, and multiple objects can maintain relative spatial relationships. Therefore, we allow multiple trajectories to control multiple objects in the image with different levels of complexity simultaneously. Our experiments validate the effectiveness of DragEntity, demonstrating its superior performance in fine-grained control in video generation.

Main paper: 
 introduction: 
 Video generation, such as text-to-video (T2V) generation [5, 7, 14] , aims to produce diverse and high-quality videos based on given text prompts. Unlike image generation [23, 25] , which focuses on generating a single image, video generation requires creating consistent and smooth motion within the generated sequence of images. Therefore, motion control plays a crucial and significant role in video generation and has received great attention in the research of various control mechanisms in recent years.Currently, in the field of controllable video generation, previous works mainly emphasized image-to-video generation, using an initial frame image as the control condition to generate video sequences, as seen in the works of [9, 19, 28] . However, relying solely on images as the control condition cannot determine the content of subsequent video frames, making it difficult to generate longer videos. As a result, research has shifted towards the text-tovideo domain, using long texts or prompt words to constrain the semantic content of video generation, as in the works of [16, 27] . Recently, some studies have utilized both text and images as control conditions for more accurate control, as demonstrated by [10, 42] . Despite this, due to the ambiguity and subjectivity of language, images can only serve as initial frames for content guidance, thus the information as control conditions remains limited in aspects such as camera movement and complex object trajectories.In the field of video generation, trajectory-based control has emerged as a user-friendly method, attracting increasing attention from researchers. CVG [13] and C2M [1] encoded images and trajectories, predicted optical flow maps and warping features as intermediate results for controllable video generation. However, warping operations often lead to unnatural distortions. To address this issue, II2VB [4] and iPOKE [3] compressed videos into a dense latent space and learned to manipulate these latent variables using RNNs. Similarly, MCDiff [34] predicted future frames in an autoregressive manner through diffusion delay. While MCDiff has shown promising results, it relies on HRNet by [33] to extract 17 key points for each individual, it can only control movements from humans, and the generated videos do not move along the prescribed trajectory paths. Moreover, MCDiff overlooks the generation of open-domain videos, significantly limiting its practical application value.As one of the representative works, DragNUWA [41] encodes sparse trajectories into dense flow space sequences, then uses them as supervisory signals to control object motion. Similarly, MotionCtrl [36] directly encodes the trajectory coordinates of each object into a vector field, using this vector field as the condition to control object motion. These works have made significant contributions to trajectory-based controllable video generation. However, can a point on the image truly represent the object?The aforementioned research still has some issues. Firstly, current works' consideration of trajectory control is not comprehensive enough. It is evident that a single pixel cannot represent an object, and dragging a single pixel cannot accurately control the corresponding object. Secondly, the objects within the pictures are quite small. Dealing with human datasets like TED-talks, Human3.6M, etc., multiple complex trajectories would only cause pixels close to the trajectory to move, thus leading to severe distortion of the human body. In fact, addressing this issue requires clarifying two concepts: 1) What is the entity. Identifying the specific area or entity to be dragged. 2) How to drag. How to achieve accurate dragging of the selected area, which means separating the background to be dragged from the foreground. For the first challenge, interactive segmentation [18, 35] is an effective solution. For example, using SAM [18] in the initial frame allows us to conveniently select the area we wish to control. For individual human twisting, fine-grained segmentation is required, such as limbs and head. In contrast, the second technical challenge poses a greater difficulty. To address this issue, this paper proposes a novel entity representation method that integrates the spatial relationship between objects to achieve precise motion control of entities in videos and generate action videos. Some works [8, 12, 29] have already demonstrated the effectiveness of using latent features to represent corresponding objects. Anydoor [8] utilizes the capabilities of Dino v2 [22] for object customization, while Video Swap [12] and DIFT [29] use the capabilities of diffusion models [23] for video editing tasks. DragAnything [39] uses diffusion models for instance segmentation to handle video generation tasks. Inspired by these works, we propose DragEntity, which leverages the latent features to represent individual entities. For the human body, this is further refined to a representation of 12 parts encompassing the torso and limbs. Additionally, we believe that the relative spatial relationship between entities is crucial for modeling the motion of objects. For instance, animals do not run into houses, and a human's arms cannot be below their legs.In our work, we utilize SVD [2] as the base segmentation model (for human bodies, LIP [11] is used for fine-grained segmentation).The training requires video data, motion trajectory points, and the entity mask of the first frame. Using the mask of each entity in the first frame, we extract the center coordinates of that entity. Then, we use CoTrack [17] to predict the motion trajectory of these points as the motion trajectory of the entity.Our main contributions are summarized as follows:• Unlike the paradigm of dragging pixels, we propose a method for dragging objects that enables true entity-level motion control and representation, ensuring the structural integrity of the object during the dragging process.• We introduce modeling of the relative spatial positions between objects to prevent the generation of highly unrealistic motion videos caused by trajectory dragging.• We have conducted experiments to validate the effectiveness of DragEntity, demonstrating its superior performance in fine-grained control over video synthesis. 
related work 2.1 image and text guided video generation: 
 Text-to-video generation has been widely studied in recent years [15, 16, 27, 37, 38] introducing text descriptions to semantically control the content of video generation. However, text alone cannot accurately describe the spatial information of visuals. Therefore, MAGE Hu et al. (2022) [38] emphasizes text-image-to-video, utilizing both semantic information from text and spatial information from images for precise video control. Similarly, GEN-1 Esser et al. (2023) [23] integrates depth maps with texts using crossattention mechanisms for control. In the domain of long video generation, text-image-to-video has also been widely used. For example, Phenaki Villegas et al. (2022) [32] generates subsequent frames by auto-regressively introducing previous frames and text, achieving long video generation. NUWA-XL Yin et al. (2023) [42] employs a hierarchical diffusion architecture to continuously complete intermediate frames based on previous frames and text. I2vgen-xl [44] introduces a cascaded network that improves model performance by separating these two factors and ensures data alignment by incorporating static images as essential guidance. Apart from academic research, the industry has also produced numerous notable works, including Gen-2 [13] [10], and SORA [6] . However, compared to the general video generation efforts, the development of controllable video generation still has room for improvement. In our work, we aim to advance the field of trajectory-based video generation.Dragging pixels leads to deformation and distortion of objects.Modeling spatial relationships, accurately controlling the movement of regions while maintaining the integrity of the human body structure. Figure 2 : Experiments on the motivation for entity representation. Existing methods (DragNUWA and MotionCtrl ) involve directly dragging pixels, which cannot precisely control the target, leading to camera motion or target structure distortion. In contrast, our method utilizes entity representation and models spatial relative positions to achieve accurate control.Early trajectory-based works [3, 4] often utilized optical flow or recurrent neural networks to achieve motion control. TrailBlazer [20] focuses on enhancing controllability in video synthesis by employing bounding boxes to guide the motion of subject. DragNUWA [41] encodes sparse strokes into a dense flow space, subsequently employing this as a guidance signal to control the motion of objects. Similarly, MotionCtrl [36] directly encodes the trajectory coordinates of each object into a vector map, using it as a condition to control the object's motion. These works can be categorized into two paradigms: Trajectory Map (point) and box representation. The box representation (e.g., TrailBlazer [20] ) only handle instancelevel objects and cannot accommodate backgrounds such as starry skies. Existing Trajectory Map Representation (e.g., DragNUWA, MotionCtrl, DragAnything) methods are quite crude, as they do not consider the semantic aspects of entities. In other words, a single point cannot adequately represent an entity. In our paper, we introduce DragEntity, which can achieve true entity-level motion control using the proposed entity representation. 
method 3.1 task definition and motivation: 
 Task Definition. The task of trajectory-based controllable video generation requires the model to generate videos based on a given image and motion trajectory. Given a point trajectory (𝑥 1 , 𝑦 1 ), (𝑥 2 , 𝑦 2 ), . . . , (𝑥 𝐿 , 𝑦 𝐿 ), where 𝐿 represents the length of the video, the conditional denoising autoencoder 𝜖 𝜃 (𝑧, 𝑐) is utilized to generate videos corresponding to the motion trajectory. In this paper, the guiding signal 𝑐 contains two types of information: trajectory points and the first frame of the video. Motivation. Recently, some trajectory control methods such as DragNUWA [41] and MotionCtrl [36] use trajectory points to control the motion of objects. These methods typically manipulate the corresponding pixels or pixel areas directly using the provided trajectory coordinates or their derivatives. However, they overlook a critical issue: as shown in Figure 1 , the pixels or pixel areas directly manipulated by the trajectory do not necessarily represent the entity we intend to control. Consequently, dragging these points does not enable object motion based on trajectory control. As illustrated in Figure 1 , we visualize the trajectory changes of each pixel in the generated video based on the co-tracker [17] . We can observe that:(1) It is evident that a single pixel or a group of pixels on an object cannot fully represent the entity (Figure 3 (b) ). From the pixel motion trajectory k of DragNUWA, it is clear that dragging a pixel of the sun does not lead to the movement of the sun; instead, it results in the camera moving upwards. This adequately demonstrates that a single pixel or a few pixels cannot represent the entire sun, and therefore, the model cannot understand the true meaning of our trajectory. We adopt an entity representation of objects as a more direct and effective way to precisely control the area we manipulate (the selected area), while the rest of the image remains unchanged.(2) As shown in Figure 1 (a), when multiple trajectories act on the same object, the motion of each part must maintain relative spatial relationships to preserve the object's structural integrity. By comparison, we observed that in videos synthesized by DragNUWA, different parts of the human body move independently under the control of trajectories, resulting in abnormal distortions. However, what we expect is for the human body to perform various combined bodily movements as a whole under the guidance of multiple trajectories, rather than each body part acting separately.Based on these new insights and observations, we propose a new entity representation that integrates the spatial relationships of objects, extracting the latent features of the objects we want to control for their representation. As shown in Figure 3 , the visualization of motion trajectories indicates that our method can achieve more precise motion control. For example, in Figure 3 (a), our method can accurately control the combined movements of turning the head and raising the hand, whereas DragNUWA only drags the corresponding pixel areas for motion, without considering the relationships between different parts, leading to abnormal appearance distortions. In Figure 3 control the rising of the sun, while DragNUWA interprets it as a camera displacement.Based on the SVD [2] model, the architecture of the model primarily consists of three parts: a denoising diffusion model (3D U-Net [24] ) that learns the denoising process in both spatial and temporal dimensions, an encoder and decoder that encode the supervisory signal into the latent space, and reconstruct the denoised latent features back into video. Inspired by ControlNet [43] , we use a 3D Unet to encode our guiding signal, then apply it to the decoder block of the denoising 3D Unet of SVD, as shown in Figure 4 . Unlike previous work, we have designed an entity representation mechanism that integrates the relative spatial relationships of objects, enabling trajectory-based controllable generation.The conditional signal of our method requires corresponding entity representations. In this section, we will describe how to extract these representations from the first frame of the image.Position-aware relation. Inspired by ParNet [40] , we propose a position-aware relationship module that simultaneously captures semantic and spatial object-level relationships. It is designed to enrich the representation of objects by adaptively focusing on the spatially relevant and semantically relevant parts of the input image.In the spatial relationship branch, for each object 𝑖, a polar coordinate system is centered at 𝑖, and the four-dimensional bounding box position 𝑝 𝑗 = (𝑥, 𝑦, 𝑤, ℎ) of object 𝑗 is transformed into a polar coordinate vector (𝜌 𝑗 , 𝜃 𝑗 ). This is because representing the spatial direction between the centers of bounding boxes 𝑖 and 𝑗 is very effective. We believe that in describing the positional relationships between objects, absolute positions are rarely used. Instead, relative positions are widely utilized (for example, the head is directly above both arms, etc.). Therefore, our method focuses on the relative positions of objects rather than their absolute positions. Low-dimensional relative positions are embedded to higher dimensions through a set of Gaussian kernels with learnable means and covariances of Gaussian distributions, where the spatial relation between objects 𝑖 and 𝑗 is easily separable. The spatial relation dimension after embedding is 𝑑 𝑝 = 64 experimentally. The kernel operator for object 𝑗 centered at 𝑖 is defined as follows:EQUATION)EQUATIONWhere 𝜌 0 and 𝜎 𝜌 are the learnable means and covariances of Gaussian distributions for relevant distance, and 𝜃 0 and 𝜎 𝜃 are the learnable means and covariances of Gaussian distributions for relevant angle.We aggregate the relative distance and angle relationship of objects 𝑖 and 𝑗 with a scaling function, which means that the strength of spatial relationships between objects can be weighted by spatial orientation. The aggregated spatial weight for the image is represented as 𝜔 𝑝 :𝜔 𝑝 = 𝑁 𝑗=1 𝜔 𝜌 𝑗 𝜔 𝜃 𝑗 𝑁 𝑗=1 𝜔 𝜌 𝑗 𝜔 𝜃 𝑗 (3)where 𝑁 is the number of objects in the image. In another branch, image features 𝑉 is linearly transformed by 𝑓 (•). Semantic relation 𝜔 𝑠 is computed as in Eq. ( 4). Dot-product attention is employed in our algorithm with a scaling factor 1√ 𝑑 𝑣, where 𝑑 𝑣 is the dimension of object features 𝑣.𝜔 𝑠 = 𝑓 (𝑉 ) 𝑇 𝑓 (𝑉 ) √ 𝑑 𝑣 (4)where 𝑑 𝑣 is the dimension of object feature 𝑉 . The intra-image relation weight 𝜔 𝐼 indicates both the semantic and spatial impact from object 𝑗. Spatial relationship 𝜔 𝑝 of different objects is fused with semantic relationship 𝜔 𝑠 between objects through Eq. ( 5). It is scaled in the range (0, 1) and can be regarded as a variant of softmax. 𝜔 𝐼 is computed as follows:𝜔 𝐼 = 𝜔 𝑝 exp(𝜔 𝑠 ) 𝑁 𝑘=1 𝜔 𝑝 exp(𝜔 𝑠 ) (5)Multi-head attention [31] is employed to adapt flexible relationships, since different heads can focus on different aspects of relation. Multiple relation features from multi heads are aggregated as follows:𝑉 𝑟 = 𝑓 (𝑉 ) + Concat[(𝜔 𝐼 𝑓 (𝑉 )) 1 , (𝜔 𝐼 𝑓 (𝑉 )) 2 , . . . , (𝜔 𝐼 𝑓 (𝑉 )) 𝐾 ] (6)𝐾 is the number of relation heads, which is typically set to be 6, same as transformer.Entity Representation. Using the image features 𝑉 , the corresponding entity embeddings can be obtained by indexing the coordinates from the segmentation mask. For convenience, average pooling is used to process the corresponding entity embeddings, resulting in the final embeddings {𝑒 1 , 𝑒 2 , . . . , 𝑒 𝑘 }, where 𝑘 represents the number of entities, and the channel size of each entity is 𝐶.To associate these entity embeddings with the corresponding trajectory points, we directly initialize a zero matrix 𝐸 ∈ R 𝐻 ×𝑊 ×𝐶 and then insert the entity embeddings based on the trajectory sequence points, as shown in Figure 5 . During the training process, we use the entity mask of the first frame to extract the center coordinates {(𝑥 1 , 𝑦 1 ), (𝑥 2 , 𝑦 2 ), . . . , (𝑥 𝑘 , 𝑦 𝑘 )} of the entity as the starting point for each trajectory sequence point. With these center coordinate indices, the final entity representation Ê can be obtained by inserting the entity embeddings into the corresponding zero matrix 𝐸 .With the center coordinates {(𝑥 1 , 𝑦 1 ), (𝑥 2 , 𝑦 2 ), . . . , (𝑥 𝑘 , 𝑦 𝑘 )} of the entity in the first frame, we use Co-Tracker to track these points and obtain the corresponding motion trajectories {(𝑥 𝑘𝑖 , 𝑦 𝑖𝑘 )} 𝐿 𝑖=1 , where 𝐿 is the length of the video. Then we can obtain the corresponding entity representation { Ê𝑖 } 𝐿 𝑖=1 for each frame. Encoder for Entity Representation. In this encoder, we utilized four blocks of convolution to process the corresponding input guidance signal, where each block consists of two convolutional layers and one SiLU activation function. Each block downsamples the input feature resolution by a factor of 2, resulting in a final output resolution of 1 8 . The encoder structure for processing the entity representation is the same, with the only difference being the number of channels in the first block, which varies when the channels for the two representations are different. After passing through the encoder, we follow ControlNet [52] by adding the latent features of Entity Representation Map Representation with the corresponding latent noise of the video:EQUATIONwhere Z 𝒊 denotes the latent noise of the 𝑖-th frame. Then the feature {R 𝒊 } 𝐿 𝑖=1 is inputted into the encoder of the denoising 3D Unet to obtain four features with different resolutions, which serve as latent condition signals. The four features are added to the feature of the denoising 3D Unet of the foundation model.During the training process, we need to generate corresponding Trajectories of Entity Representation and 2D Gaussian, as shown in Figure 5 . First, for each entity, we calculate its incircle circle using its corresponding mask, obtaining its center coordinates (𝑥, 𝑦) and radius 𝑟 . Then we use Co-Tracker to obtain its corresponding trajectory of the center {(𝑥 𝑖 , 𝑦 𝑖 )} 𝐿 𝑖=1 , serving as the representative motion trajectory of that entity. With these trajectory points and radius, we can calculate the corresponding Gaussian distribution value at each frame. For entity representation, we insert the corresponding entity embedding into the circle centered at (𝑥, 𝑦) coordinates with a radius of 𝑟 . Finally, we obtain the corresponding trajectories of Entity Representation and 2D Gaussian for training our model.Loss Function. In video generation tasks, Mean Squared Error (MSE) is commonly used to optimize the model. Given the corresponding entity representation Ê and 2D Gaussian representation 𝐺, the objective can be simplified to:EQUATIONwhere 𝐸 𝜃 denotes the encoder for entity and 2D Gaussian representations. 𝑀 is the mask for entities of images at each frame. The optimization objective of the model is to control the motion of the target object. For other objects or the background, we do not want to affect the generation quality. Therefore, we use a mask 𝑀 to constrain the MSE loss to only backpropagate through the areas we want to optimize.Inference of User-Trajectory Interaction. During the inference process, users simply need to click to select the area they want to control with SAM [18] . For the human body, this is further refined into segmented areas for each part. Then, by dragging any pixel within that area, they form a reasonable trajectory. Our model can then accurately control the area based on this trajectory to generate a video corresponding to the desired motion. 
experiments 4.1 experiment settings: 
 Implementation Details. All our training is based on the stable Video Diffusion (SVD) [2] architecture and weights, which have been trained to generate 20 frames at a resolution of 320×576. All experiments were conducted on PyTorch, using Tesla A100 GPUs. AdamW was used as the optimizer for a total of 100k training steps, with a learning rate of 1e-6.Evaluation Metrics. We adopt two types of evaluation metrics: 1) Video Quality Assessment: We use the Frechet Inception Distance (FID) and Frechet Video Distance (FVD) [30] to evaluate the visual quality and temporal coherence. 2) Object Motion Control Performance Evaluation: The Euclidean distance between the predicted object trajectory and the ground truth trajectory (ObjMC) is used to evaluate the object motion control. Furthermore, in user studies, we randomly generated 50 videos and had 10 non-professional individuals vote on the quality of the generated videos and the motion trajectories.Datasets.We adopt VIPSeg [21] , WebVid [22] and TED-talks [26] as our testsets. Since FVD requires videos to have the same resolution and duration, we resized the VIPSeg validation dataset to 256×256 and reduced its length to 14 frames for assessment. We use the VIPSeg and WebVid as our training sets, and employed a collaborative tracker to obtain the corresponding motion trajectories, which served as annotations.Comparisons with State-of-the-Art Methods. Video Quality Evaluation on on the WebVid datasets. Table 1 presents a comparison of video quality on the WebVid datasets using the FID metric. We controlled for other conditions being the same (basic architecture) and compared the performance between our method and DragNUWA. Our FID score reached 34.5, significantly outperforming the current SOTA model DragNUWA (34.5 vs. 36.9) . According to MotionCtrl [36] , motion control performance is evaluated using ObjMC by calculating the Euclidean distance between predicted trajectories and ground truth trajectories. Compared to DragNUWA, our model achieved state-of-the-art performance with a score of 302.7. Additionally, FVD assesses the temporal coherence of generated videos by comparing the feature distribution between generated videos and ground truth videos. Compared to the performance of DragNUWA (521.7), our model also achieved superior temporal coherence with a score of 510.8, significantly improving by 10.9.At the same time, we also conducted visual comparisons on the TedTalk and VIPSeg datasets. As shown in Figure 5 , we can observe that when DragNVWA uses multiple trajectories to control the motion of characters, it results in distortion (third and seventh rows), artifacts (fifth row), and multiple hands (first row). As shown in Figure 6 , when DragNVWA controls multiple objects, there are issues such as appearance distortion (third row), incorrect movement directions (first row), and incorrect camera movements (fifth row), but our model DragEntity can precisely control motion. 2 shows the ablation studies of these two aspects. To study the influence of entity representation, we observe performance changes by determining whether this representation is included in the final embedding. Since the entity representation primarily affects the object motion in generated videos, we only need to compare ObjMC, while FVD and FID metrics focus on temporal consistency and overall video quality. With entity representation, the model's ObjMC saw significant improvement, reaching 311.4. Similar to entity representation, we observe changes in ObjMC performance by determining whether the final embedding includes spatial position relationships. The spatial relationships between objects led to a performance improvement of 29.4, reaching 339.3. Overall, the highest performance occurs when both entity representation and spatial position relationships are used together, achieving 302.7. This phenomenon indicates that these two representations have a mutually reinforcing effect, contributing to the precise control of the trajectory.DragNVWA Ours DragNVWA Ours Ours Ours DragNVWA DragNVWAAdditionally, we also explored the impact of the loss mask. Table 3 presents the ablation study for Loss Mask. When the loss mask is not used, we directly optimize the MSE loss for each pixel of the entire image. The loss mask can bring certain benefits, approximately improving ObjMC by 13.4.User Evaluation We conducted a user survey to assess video authenticity from MotionCtrl, DragNVWA, and our method, each at 576×320 resolution. Ten volunteers chose the best method based on: 1) structural integrity; 2) trajectory consistency; 3) overall quality.  
conclusion: 
 In this paper, we introduce a new trajectory-based motion control method and present two new insights: 1) Pixel points controlled by trajectories do not adequately represent entities. 2) When multiple trajectories act on the same object, the motion of each part must maintain relative spatial relationships to preserve the object's structural integrity. To address these two technical challenges, we propose DragEntity, which utilizes latent features to represent each entity. Our proposed entity representation, which incorporates spatial relative position relationships as a self-domain embedding, can control the motion of entities in the image while maintaining structural integrity. Experiments validate the superiority of our method over existing approaches, demonstrating its ability to effectively generate fine-grained videos.  
 
 Appendix:  


Decision: The authors introduces DragEntity, a user friendly interaction method,  to control the motion of multiple objects in a video using entity representation while preserving spatial relationships. The authors demonstrate the superior performance through experiments. 

The reviewers identify several strengths of the proposed method along with some weaknesses (see details in the reviews). The authors agree on the suitability of the paper for the ACM MM conference. While some concerns from the reviewers remain after the rebuttal, the quality of the paper is suitable enough to be accepted provided some of the limitations are addressed by the camera ready version.

Reviews: review: paper_summary: paper_summary: This article introduces a video generation model called DragEntity, which uses entity representation to control the motion of multiple objects in a video. Compared to previous methods, DragEntity offers a user-friendly interaction method and can simultaneously and accurately control multiple objects moving along complex trajectories while maintaining their relative spatial relationships. Experiments have verified its effectiveness and superior performance in fine-grained control for video generation.
strength_weakness: strength_weakness: 1. DragEntity allows users to interact with the video generation model by drawing trajectories, simplifying the user operation process.
2. Unlike pixel-based methods, DragEntity achieves true entity-level motion control, ensuring the structural integrity of objects during the dragging process.
3. Unlike pixel-based methods, DragEntity achieves true entity-level motion control, ensuring the structural integrity of objects during the dragging process.
4. DragEntity introduces modeling of relative spatial relationships between objects, preventing the generation of highly unrealistic motion videos caused by trajectory dragging.
5. The model can accurately control the motion of regions rather than camera motion, avoiding deformation or distortion of objects that can result from direct pixel dragging.
limitations: limitations: 1. Judging from the results, the method's dragging of human bodies appears to be quite eerie. The quality of generation does not seem to be significantly better. Has the author provided an explanation for this issue, as it does not substantiate the contribution points proposed in the paper?
2. The discussion of the method in the paper seems rather limited in the experiments. Has the author further explored the model's generalization ability, computational cost, and the finesse of control?
paper_summary: paper_summary: This paper proposes a method for trajectory guided video generation with a new entity representation. It first extracts the entity representation with a proposed module, then the proposed main framework will utilize the entity representations to precisely control motion in the generated video based on a previous model SVD.
strength_weakness: strength_weakness: The entity representation designed to enrich the representation of objects by adaptively focusing on the spatially relevant and semantically relevant parts of the input image is novel.

With the new entity representation, authors show this method can perform better on issues like deformation and distortion of objects, camera movement, etc.
limitations: limitations: Authors show several cases that the proposed method can perform better than other methods:

For the camera movement issue, the comparison with DragNVWA may not be fair. If I understand correctly, proposed method is based on SVD segmentation, which already implicitly contains rich segmentation information beyond the training data of DragNVWA, while DragNVWA is just using optical flow within the training data. DragNUWA’s camera movement problem in Fig. 2 may because there’s few sun rise videos in the training dataset, mostly camera movement. But your method is able to do it may because the SVD can segment the sun. Is the proposed entity representation with spatial relationships play a key role here? What if DragNVWA + SVD segmentation? Ablation study may be needed.

For the object deformation/distortion issue, I’m not convinced why the designed method can resolve this issue. Like Fig. 2, the deformation/distortion is just in the head area, maybe proposed method can segment the head with the LIP, but how can the relationships between objects/components help resolve deformation/distortion within one object/component?

The main issue of this paper to me is, the reason why the proposed method can solve the issues is not fully explained with descriptions/experiments.

Another issue is, the terminology is not consistent, which may lead to confusion. For example, in Sec. 3.2 it says “Entity Representation includes Spatial Relationships”, but in the ablation study, it says “Entity representation and Spatial Relationship are core components of our work”, I’m not sure how you do the experiment only with spatial relationship.

Also, the experiments may not be enough. Only table 1 is the comparison table with other method on 1 dataset, more comparison will be more convincing.


Other minor issues/comments:

In abstract, MotionCtrl in line 17 should be DragEntity?

Fig 6, last image of the 4th row, should show the optical flow for your method, here seems you are using the original one?

Fig 6, first and second row comparison, DragNVWA’s movement direction looks ok to me (maybe not as much as your method), and DragNVWA actually has less distortion (number of human, shape of human), so I would say DragNVWA looks better in this case.
paper_summary: paper_summary: This paper tackles controllable video generation via entity representation to control the motion of multiple objects. To ensure more user-friendly interaction, the proposed DragEntity method utilizes trajectories to control objects in the image. The method is compared with previous work DragNUWA. The results on object moving shows that DragEntity is able to perform fine-grained control with better performance compared with DragNUMA and MotionCtrl in terms of several evaluation metrics. However, additional clarification along with experimental comparisons may be needed for better evaluations of the proposed video generation method.
strength_weakness: strength_weakness: 1. DragEntity focuses on trajectory-based control of video generation, which has emerged to be user-friendly regarding content guidance for movement. By leveraging the latent features to represent individual entities, DragEntity considers entity-level motion and relative spatial positions between objects to ensure the structural and motion integrity of the moving object.
2. Multiple evaluation metrics are involved to evaluate the quality of generated videos from different perspectives.
limitations: limitations: 1. In Section 4.1, the “Datasets” paragraph mentions that “VIPSeg[21], WebVid[22] and TED-talks[26]” are utilized as test sets, but only quantitative results on WebVid are given as in Table 1. What about the results on the other two datasets? 
2. Figure 6, the sun disappears in the last example for the proposed method. Since this example focuses on the movement of the sun, how does the disappearance of the moving objects demonstrate the performance of DragEntity?
3. Could DragEntity be compared with methods that are able to perform image editing that includes object moving, e.g., DragGAN and DragDiffusion? 
4. The first sentence of the paragraph above the itemized contributions in the introduction mentions SVD [2] as “the base segmentation model”. This description may require a clarification since SVD is known as Stable Video Diffusion model for video generation, not for segmentation. For the extraction of entity representation, how the mask of each entity is acquired? If the DragEntity method assumes a given mask, it may be unfair to compare this with existing video generation methods.
paper_summary: paper_summary: The paper proposes to improve the trajectory guided video generation by using entity representations, rather than point/pixel/bounding box representations. Specifically, the paper utilizes stable video diffusion and CoTrack to segment the frames to identify some entities and track their motion trajectories, which allows users to drag them to control the video generation. Meanwhile, the proposed method also employs a position-aware relation module to model and constrain the relative positions or motion among multiple object parts or entities. The proposed method has been evaluated on 3 public datasets and compare favorably with the SOTA.
strength_weakness: strength_weakness: The paper is well motivated since how to control the video generation by the motion of multiple objects is a critical problem for users to adopt the video generation technology.

The proposed entity representation and position-aware relation module are new and sound approaches to advance the trajectory guided video generation.

The experiments are convincing to demonstrate the clear advantage of the proposed method.

Overall, this is a descent paper which is a clear step to advance  trajectory guided video generation.
limitations: limitations: I wonder how sensitive of the RBF kernel and the learnable means and covariances, as well as the semantic relation in Eq.4 to the video generation performance. Even for the same kind of objects, the motion or relative positions among the parts could be dramatically different due to the camera view angles. Please elaborate on the generalizability of the position-aware relation, since the training videos may not cover all possible motion and view angles of objects.

Typo: "on on" line 641.