HeroMaker: Human-centric Video Editing with Motion Priors

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 17:19:47

Abstract: Video generation and editing, particularly human-centric video editing, has seen a surge of interest in its potential to create immersive and dynamic content. A fundamental challenge is ensuring temporal coherence and visual harmony across frames, especially in handling large-scale human motion and maintaining consistency over long sequences. The previous methods, such as diffusion-based video editing, struggle with flickering and length limitations. In contrast, methods employing Video-2D representations grapple with accurately capturing complex structural relationships in large-scale human motion. Simultaneously, some patterns on the human body appear intermittently throughout the video, posing a knotty problem in identifying visual correspondence. To address the above problems, we present HeroMaker. This human-centric video editing framework manipulates the person's appearance within the input video and achieves inter-frame consistent results. Specifically, we propose to learn the motion priors, transformations from dual canonical fields to each video frame, by leveraging the body mesh-based human motion warping and neural deformation-based margin refinement in the video reconstruction framework to ensure the semantic correctness of canonical fields. HeroMaker performs human-centric video editing by manipulating the dual canonical fields and combining them with motion priors to synthesize temporally coherent and visually plausible results. Comprehensive experiments demonstrate that our approach surpasses existing methods regarding temporal consistency, visual quality, and semantic coherence.

Main paper: 
 introduction: 
 Human-centric video editing focuses on modifying the individual within a given video and generating temporally coherent results. This technique has numerous potential applications, such as media content production, virtual reality, and video games. A pivotal challenge in human-centric video editing is maintaining coherence and harmonious results across frames when people can move freely in the video.Recent diffusion-based video editing explores extracting and incorporating various structural correspondences using infer-frame attention maps [6, 8, 32, 55] , optical flows [11, 58] and nn-fields [19] . Although the temporal consistency has improved, it still struggles with flickering and length limitations. Alternatively, some researchers have explored video-2D representations, storing the information of the video in atlases [30] or canonical images [46] to propagate changes over time. However, it grapples with accurately capturing complex structural relationships in large-scale human motion. Moreover, many studies have focused on reconstructing a human body in 3D and attempting to edit it. While promising, these methods often present challenges regarding cost, size, and unfriendly user-controlled environments due to the semantic-less texture maps and the requirement for long-term optimization processes.Since each human part is unique, patterns on the human body appear intermittently throughout the video due to self-occlusion, posing a knotty problem in identifying correspondence to ensure consistent video editing. As a method of video-2D representation, CoDeF [46] builds correspondence via learning a neural deformation field from a canonical image to each frame and does the video editing on the canonical field, which improves the correspondence between frames. Although it achieves high-fidelity reconstruction, the canonical image differs from natural images, leading to difficulties when editing with image editing tools, including ControlNet [62] , and resulting in editing challenges to generate semantically plausible results.In light of CoDeF's success, we leverage the human body mesh to obtain semantic human canonical images, which provide structural and texture correspondence in 3D space. Our framework defines motion priors, incorporating human motion warping, neural margin refinements, and dual canonical fields to achieve this goal. To obtain the motion priors from a given video, our model employs an off-the-shelf human mesh estimator to set up an initial body mesh. Then, we refine the body mesh in a two-step optimization to close its shape to the person's in videos to ensure more accurate human motion warping. Resorting the motion priors, our model defines the dual canonical fields with a frontal and back body mesh under the A-pose to obtain the vast majority of human body information. Subsequently, it reconstructs the video with explicit human motion warping and neural margin refinements between dual canonical fields and each human-centric video frame. Additionally, our model supports diverse user interactions for modifying the videos. Hero-Maker performs human-centric video editing by manipulating the semantic-aware dual canonical fields. Together with the motion priors, it synthesizes temporally coherent and visually plausible results.We summarize our contributions as follows:• We propose a new human-centric video representation combining motion priors and deformation fields to reconstruct and edit the video. Recent works attempt to extend a latent diffusion model into a T2V editing model [2, 5, 6, 15-17, 19, 21, 22, 24, 25, 27, 32, 34, 35, 38, 41, 44, 48, 49, 52, 55, 56, 59-61, 65] . Tune-A-Video [55] and Control-A-Video [8] extend a latent diffusion model to the spatial-temporal domain and finetune it with source videos. However, they still have difficulties in modeling complex motions and long sequences. Text2Video-Zero [32] and ControlVideo [64] use ControlNet [62] to preserve the per-frame structure but struggle to temporal consistency. FateZero [48] and vid2vid-zero [53] use attention maps to enhance shape-aware editing based prompt-to-prompt [23] , but they still have temporal issues. Rerender-A-Video [58] , TokenFlow [19] ,and VideoControlNet [27] utilize optical flow to control inter-frame relationships to improve consistency. However, they still face challenges when addressing large-scale human motion and rotation issues. TokenFlow [19] enforces linear combinations between diffusion features based on source correspondences. However, the pre-defined combination weights are not adapted to all videos, resulting in high-frequency flickering. Because TokenFlow [19] needs to cache information for each frame when processing long-sequence videos, resulting in insufficient memory, we will not compare it with this method. The above methods explore the augmentation of inter-frame attention modeling on a diffusion model. They ensure the correct spatial structure but still challenge temporal consistency. Recently, AnimateDiff [21] presented a motion module trained on extensive video data without a fineturing diffusion model, improving temporal consistency. Furthermore, human-centric videos have further explored some workss [26, 57] and achieved visually plausible results. However, generating the same effect in videos in a small amount of video data poses challenges. Unlike these works, our method leverages human motion priors to achieve text-guided video editing effectively. 
temporal propagation in video editing.: 
 Another significant line of video editing work relies on a powerful video representation. VideoSnap [66] compresses videos using spatio-temporal feature maps into one or several images and then trains an expansion network to transform these images back into videos. The layered neural atlas [30] factorizes the input video using a layered presentation. It maps the subject/background of all frames using 2D UV maps as an intermediate editing representation. Once the layered neural atlas is learned, editing can occur either on keyframes or on the atlas itself, and the editing results consistently propagate to other frames [4, 7, 13, 28, 33] . CoDeF [46] incorporates the 3D deformation field with the 2D hash-based canonical image to improve the video representative capability further. However, Video Snapshot [66] , atlas [30] , and canonical image [46] all utilize optical flow to help to predict the relationships between each frame. They encounter difficulties in reconstructing and editing videos with large-scale human motion. This leads to incorrect correspondence and texture information, resulting in unnatural results.3D human reconstruction and editing are closely related to humancentric video editing tasks. Many papers aim to reconstruct an accurate human body and texture using the SMPL+D model [1, 18] or implicit functions [20, 29, 54] through monocular videos. However, most focus on reconstructing and driving more accurate human models without considering editing effects and friendliness. Some works attempt to edit in 3D, SINE [3] and SKED [42] support editing a local region of the base NeRF [43] . Dyn-E [63] and Control4D [51] propose to edit the contents of dynamic NeRFs. However, Dyn-E [63] can only edit the local appearance with user manipulation. Control4D [51] needs multi-view videos as input and can only handle videos with small motions and short video lengths. Recently, DynVideo-E [37] has attempted to edit monocular videos in 3D through text. However, it is not user-friendly for reconstruction and editing operations to take tens of hours to complete. Although these methods can produce high-fidelity results, their cost, size, and controlled environment are unfriendly to users. Instead, video or image editing frameworks are more likely to avoid these shortcomings. Our method introduces the motion priors to the human body. Then, we convert it into pixel position relationship conversion between each frame. While retaining the correctness of the 3D structure, it transforms the task into 2D image editing, which is also one of the primary motivations of our work. Our work focuses on proposing a new human-centric video representation to solve problems with the image or video editing task. Meanwhile, we will not compare these methods. [37, 51, 63] . due to some recent work not being open-source yet. 
method: 
 Given a human-centric video, we aim to modify its visual attributes based on diverse user interactions while maintaining correct structural correspondence and temporal consistency. We tackle this problem with a multi-stage framework, namely reconstructing and editing. As shown in Fig. 2 , HeroMaker introduces a novel video representation by leveraging the motion priors based on the SMPL-X [47] , which establishes the transformation correspondence from the canonical field to each video frame. In the following, we first illustrate the motion priors in Sec.3.1, and then our novel video representation is elaborated in Sec.3.2, followed by details of the editing procedure and applications of the whole framework in Sec.3.3.As previously stated, the visual quality of video editing largely depends on the established video representation. To learn a better video representation for facilitating subsequent video editing, we resort to readily accessible motion priors. Specifically, we mitigate the deformation ambiguity by breaking it down into two components: known human motion warping and neural margin refinement. Thus, our first step is to obtain reliable motion priors. As depicted in 2 (a), starting with a human-centric video {𝐼 𝑖 } 𝑁 -1 𝑖=0 consists of 𝑁 frames, we apply the off-the-shelf OSX [36] to predict the camera parameters 𝑃 𝑖 and SMPL-X [47] coefficients due to its robustness toward partial observations and high efficiency. SMPL-X [47] is defined as a differentiable function 𝑆 (𝛽, 𝜃,𝜓 ) → (𝑉 , 𝐹 ) that outputs a 3D human body mesh with 10475 vertices 𝑉 ∈ R 10475×3 , 20908 faces 𝐹 ∈ R 20908×3 , where 𝜓 ∈ R 10 is the facial expression parameters, 𝛽 ∈ R 10 and 𝜃 ∈ R 22×3 are the body shape parameters and pose parameters, respectively. In order to enhance the accuracy of the transformation correspondence, instead of directly utilizing the regression-based estimation provided by OSX [36] , we adopt a two-step optimization strategy to obtain a more accurate SMPL-X [47] fit. Firstly, we refine the SMPL-X coefficients with 2d keypoints. Specifically, we leverage mmpose [12] to attain 2D keypoints 𝑃 𝑖 (2D) for each frame 𝑖. We optimize over the learnable parameters 𝜃 𝑁 -1 𝑖=0 by minimizing the difference between estimated 2D keypoints 𝑃 𝑖 (2D) and corresponding projected 2D joints 𝑃 𝑖 (𝐽 sub ), where 𝑃 𝑖 is the projection matrix. Additionally, we employ a temporal regularization term L 1 reg on output mesh vertices 𝑉 𝑖 𝑚 to ensure continuity. The optimization objective of the first stage is:EQUATION)L kps = ∥𝑃 𝑖 (2D) -𝑃 𝑖 (𝐽 sub )∥ 2 2(2)EQUATION)To further improve the flexibility of the SMPL-X [47] model's expression ability, making it able to match the clothed human better in the video, rather than a skinned person. In the second stage, we added a per-vertex offset 𝐷 ∈ R 10475×3 , to capture the details of each frame and define the model as:𝑆 (𝛽, 𝜃,𝜓, 𝐷) = LBS(𝑇 (𝛽, 𝜃,𝜓, 𝐷), 𝐽 (𝛽), 𝜃,𝑊 )𝑇 (𝛽, 𝜃,𝜓, 𝐷) = 𝑇 (𝛽, 𝜃,𝜓 ) + 𝐷 (5)EQUATION)where T is a mean shape template, 𝐵 𝑠 , 𝐵 𝑝 and 𝐵 𝑒 are shape, pose and expression blend shapes, respectively. LBS denotes linear blending skinning and 𝑊 is the vertices' skinning weights. We only optimize 𝐷 in this stage to avoid overfitting. Since we expect the rendered human outline to align with the foreground mask, we utilize mask loss as a supervision. We use SAM-Track [10] to get a per-frame binary mask 𝑀 𝑖 of human and minimize the difference between the silhouette of the rendered body 𝑅 𝑠 (𝑃 𝑖 , {𝑉 𝑖 , 𝐹 }) and the obtained mask 𝑀 𝑖 , where 𝑅 𝑠 denotes the differentiable silhouette rasterizer. To ensure mesh smoothness, we regulate the offset with Laplacian smoothing loss [14, 45] 𝐿 Laplacian and 𝐿 2 regularization. The optimization objective of the second stage is:EQUATION)EQUATION)L 2 reg = 𝐿 Laplacian (𝐷) + 𝛾 ∥𝐷 ∥ 2 2 (9)Finally, we acquire motion priors that is sufficiently expressive for the video.We find a powerful video presentation with better temporal continuity than the inter-frame attention model. With the motion priors described in Sec. 3.1, we target a more editing-friendly video representation, which could effectively convert the human-centric video editing problems into image editing problems. While overfitting the observed video using neural representation is relatively accessible, it often leads to a noisy canonical field due to the ill-posed nature of solving the deformation field. Intuitively, the prerequisite for promising editing is establishing a meaningful canonical field. Meanwhile, a well-defined deformation field can relieve the ambiguity in the canonical field, subsequently benefiting high-quality editing outcomes. The previous methods were mainly divided into two types. The first type [30] uses a UV mapping relationship between pixel space and layered neural atlas, which caused inconvenience during editing. The second type [46] wanted to compress video content onto images, but finding the correspondence between frames in large-scale human motion is challenging. Thus, we devise the canonical fields and decompose the temporal deformation in video based on the motion priors. As shown in 2, our video representation comprised of three components: Dual canonical fields. We define the canonical human body as the A-posed SMPL-X+D, 𝑆 𝑐 = 𝑆 ( β, 𝜃 𝐴 , ψ ) + D with mean estimated coefficients across video frames. Specifically, we adopt a dual canonical fields design in which we choose the front view 𝐶 front and back view 𝐶 back of the canonical human body for information complementarity. As for the network structure, our dual canonical fields are constructed using two 2D multi-resolution hash encodings, which map a 2D position (𝑥, 𝑦) to (𝑅, 𝐺, 𝐵) color. Human motion warping. To alleviate the issue of overfitting resulting from directly learning a deformation field [46] , we expect that explicit human motion warping dominates the overall deformation, while neural deformation serves as a refinement. Human motion warping is parametric-free, yet it provides semantic correspondences across frames in video representation. Inspired by Liquid Warping GAN [39, 40] , we build human motion warping using the Neural Mesh Renderer (NMR) [31] . To reconstruct a target frame 𝐼 𝑖 , we first query the canonical fields to acquire two canonical images 𝐼 front and 𝐼 back , and then we embed them into texture space using a weak-perspective camera as 𝑆 𝑐 . Since our motion priors are topologically consistent, we can easily obtain the transformation 𝑇 tr front and 𝑇 tr back , which warp the information from the canonical fields to the target frame. Moreover, we compute a mask to fuse the information from 𝐶 front and 𝐶 back . For more details, please refer to the supplementary materials. It is worth noting that we define the deformation in a canonical-toobservation direction, which naturally prevents the drawback of backward deformation [9] . Neural Margin refinement. Now, we can make approximate transformations with human motion warping. However, be aware that human motion warping only accounts for rigid transformation, insufficient for clothes and fine-grained non-rigid deformations. In other words, the information in the position 𝐼 𝑖 (𝑥, 𝑦) is not solely determined by the transformation relationship 𝑇 𝑡𝑟 𝑓 𝑟𝑜𝑛𝑡 |𝑏𝑎𝑐𝑘 . To this end, we design a small refinement field and padding strategy to refine the margin part of the human in a video.As for the refinement field, we implement 2D multi-resolution hash encoding as the backbone. Specifically, we feed in a triplet (𝑥, 𝑦, 𝑖) into the refinement field and produce the residual Δ𝑇 : (Δ𝑥, Δ𝑦). Formally, the full transformation from front or backcanonical image to each target frame 𝑇 = 𝑇 tr front|back + Δ𝑇 , where 𝑇 tr front|back is estimated based on SMPL-X [47] .As shown in Fig. 4 , we set the final transformation relationship to be related to the transformation relationship obtained by the human motion warping and margin refinement module.For points within the transformation relationship 𝑇 𝑡𝑟 𝑓 𝑟𝑜𝑛𝑡 |𝑏𝑎𝑐𝑘 , we set the full transformation relationship 𝑇 𝑖𝑛 as the result of the current point's transformation relationship 𝑇 𝑡𝑟 𝑓 𝑟𝑜𝑛𝑡 |𝑏𝑎𝑐𝑘 (𝑖𝑛) and add the result Δ𝑇 𝑖𝑛 refined by the margin refinement module.EQUATION)For points outside the transformation relationship 𝑇 𝑡𝑟 𝑓 𝑟𝑜𝑛𝑡 |𝑏𝑎𝑐𝑘 and inside the human mask 𝑀, we set the full transformation relationship 𝑇 𝑜𝑢𝑡 as the transformation relationship of the point nearest to the effective transformation matrix 𝑇 𝑓 𝑟𝑜𝑛𝑡 |𝑏𝑎𝑐𝑘 (𝑛𝑒𝑎𝑟𝑒𝑠𝑡 -𝑖𝑛) , and add the result Δ𝑇 𝑜𝑢𝑡 refined by the margin refinement module.EQUATIONWe select the two frames 𝐼 nn_front , 𝐼 nn_back from the video closest to the front-view and back-view canonical field as regularization for training. Specifically, we additionally reconstruct the target frame Î ′ 𝑖 with the information from 𝐼 nn_front or 𝐼 nn_back according to the orientation similarity of the pelvis. In this way, we can largely preserve the semantic information. Our representation is jointly trained by minimizing the reconstruction loss 𝐿 rec between the predicted image Î and the original one 𝐼 using mean square error. Moreover, we constrain the output of the deform field using 𝐿 2 norm empirically. Overall, the loss function of video reconstruction with motion priors can be written as:EQUATION)EQUATIONUpon the optimized video representation, we can obtain the trained front canonical image 𝐼 front and back canonical image 𝐼 back by querying the 𝐶 front and 𝐶 back with position (𝑥, 𝑦). Although our dual canonical fields design retains semantic and structural information, they also introduce a new challenge in terms of semantic consistency. For the randomness during the diffusing and denoising process, editing two canonical images separately using Control-Net [27] may yield inharmonious results. We suggest resolving this issue through a simple yet effective strategy to ensure editing coherence. Specifically, we concatenate the 𝐼 front and 𝐼 back along the width axis and feed it into ControlNet [27] . The self-attention mechanism implicitly builds the correlation between two canonical images.As shown in Fig. 2 (c), we explore two different editing scenes: 1) Video editing by prompt. Users could modify the content of the human and background separately using text inputs. 2) Video editing with user input. Users could directly draw on the canonical images at their will. For example, they can attach a logo to their clothes and automatically propagate it throughout the video.Moreover, HeroMaker supports editing a person individually within a multi-person video, which differs from most competing methods. 
experiments 4.1 experimental setup: 
 Implementation Details. HeroMaker is implemented in PyTorch. In the first stage, we optimize the mesh with the Adam optimizer(𝑙𝑟 = 0.0001, 𝛽 = (0.9, 0.99)) for 200 iterations. The regularization parameter, denoted as 𝜆 1 𝑟𝑒𝑔 = 0.2. In the second stage, We optimize for 20 iterations per frame with a learning rate of 0.0005, 𝜆 2 𝑟𝑒𝑔 = 0.2 and 𝛾 = 10. During video reconstruction, we jointly trained dual canonical fields and neural margin refinement field together with the Adam optimizer(lr=0.0001, 𝛽 = (0.9, 0.99)) for 15000 iterations. We employ the pre-trained Stable Diffusion v1.5 model, and Con-trolNet [62] provides structure guidance regarding edges. For image editing, we implement 30 timesteps for DDIM sampling. Dataset. We validate the effectiveness of our full pipeline using two datasets, including selected videos from the iPER [39, 40] Baselines. We compare our method with five baselines: NLA [30] , Text2Video-Zero [32] , Rerender-A-Video [58] , StableVideo [7] and CoDeF [46] . We compare our method with NLA [30] and CoDeF [46] in video editing with the user input task to validate the ability of the model to represent the video in a structure-aware correspondence.For video editing by prompt task, we compare our model with Text2Video-Zero [32] , Rerender-A-Video [58] , StableVideo [7] and CoDeF [46] to show the temporal consistency and ability to match the prompts. Evaluation Metrics. Human-centric video editing aims to faithfully reflect the editing prompt while maintaining original shape coherency and temporal consistency. We further propose to measure the shape coherency. In detail, we estimate the human mesh in the original and edited video using OSX [36] and compute the average mesh vertices error as 𝐸 vertices . For textual alignment, we report the average CLIP [50] score, which computes the cosine similarity between the prompts' CLIP [50] embedding and each frame's image embedding in the edited video. However, simply using these metrics cannot fully represent the visual quality of edited videos. We thus conducted a user study. We show the textual descriptions and editing results of different methods, asking them to rate in three aspects: textual fidelity with temporal continuity, shape preservation, and comprehensive visual effect.Quantitative Comparison. Following previous works, we evaluated our method and baselines with different metrics. As indicated in Table . 1, our method surpasses previous works in all metrics, demonstrating that our editing results align closely with the prompts and maintain the original body shape. We further conduct user studies as described in Sec.4.1. As shown in Table . 2 and Table . 3, the participants exhibit a clear preference for our results.CoDeF NLA Input Ours Our method supports local editing and allows users to add customized icons accurately into the region of interest. We compare our method against NLA [30] and CoDeF [46] .Qualitative Comparison. Fig. 3 presents the visual results of promptbased video editing. Text2Video-Zero [32] and Rerender-A-Video [58] generate outputs semantically aligned with the text description but fail to maintain temporal consistency. For instance, the body shapes are flickering, and the arms are distorted (see Spider-Man and Superman in Fig. 3 ). StableVideo [7] exhibits satisfactory temporal consistency. However, it is prone to generating outputs with reduced fidelity. Since CoDeF learns the deformation field and the canonical image without structure information, it generates different results from natural images when handling human-centric videos. According to Fig. 3 , by leveraging the motion priors, our method successfully achieves temporal consistency while preserving fidelity. Furthermore, representing the video with motion priors allows our model to edit the human body locally. It enables users to edit regions of interest while maintaining the other parts. In Fig. 5 , we compare our method to NLA [30] and CoDeF [46] in user interactive video editing. NLA [30] and CoDeF [46] models use optical flow to maintain the correspondence between frames. However, estimating optical flow for complex human motion is difficult, which leads to visual flaws. Although NLA [30] shows good textured results, it fails to maintain geometry consistency between human motion. CoDeF [46] leads to information losses when encoding the video into a canonical content field. In some cases, the correspondence of body deformation deviates, causing unpleasant editing results. In contrast, our model utilizes motion priors and thus learns human-aware canonical fields, ensuring that the editing contents are attached to the appropriate positions.Input Reconstruction Edited VideosAdditionally, unlike most previous methods, HeroMaker offers the ability to easily modify any character within a video containing multiple people, as illustrated in Fig. 6 . This capability enhances the appeal and flexibility of human-centric video editing, providing users with a unique and engaging experience. Compared to method (a), method (b) benefits from a neural deformation module to correct estimation errors from the SMPL-X network. The method (c) further improves the results using first-stage SMPL-X refinement that effectively improves motion priors. Our full uses a neural deformation module and two stages of SMPL-X refinement to achieve precise mesh deformation. Additionally, we demonstrate the necessity of learnable canonical fields by comparing method (e) versus method (f).To verify the contributions of different modules to the overall performance, we systematically deactivate specific modules in our framework and present the visual comparison in Fig. 7 . In this section, we mainly analyzed the impact of varying degrees of SMPL-X refinement, whether to add a deform module and the necessity of learning canonical fields. We define method (a) as the model without SMPL-X refinement and the neural deformation module. Method (b) incorporates the neural deformation module into the baseline. Considering that our framework performs SMPL-X refinement in two stages, method (c) applies only the first refinement stage, whereas method (d) represents our full model. Neural deformation module. The neural deformation module aims to correct estimation errors from the SMPL-X network by ensuring the alignment of frame images with the canonical fields under motion priors. The comparison between methods (a) and (b) demonstrates improvements in clarity and reduction in edge deviations, highlighting the module's efficacy in enhancing visual quality by learning accurate correspondences.SMPL-X refinement. The refinement process mitigates mesh deviations detected by the SMPL-X network and utilizes 2D keypoints and inter-frame mesh deviations for optimization in the first stage. It is visible that the improvement in image quality in method (c) indicates that the correctness of motion priors has a positive impact on the results. The second refinement stage further rectifies edge artifacts, underscoring the refinement's critical role in achieving precise mesh deformation.Learnable canonical fields. Additionally, to demonstrate the necessity of learnable canonical fields, we select the two images closest to the front and back view as canonical images and then optimize the deform network to obtain the final results. We believe that this can extract as much information as possible from the video while ensuring semantic information. However, as shown in Fig. 7 , by comparing method (e) versus method (f), we observe that learnable canonical fields capture more detailed and relevant information from video sequences, thereby reducing reconstruction errors and improving edge definition. Constant canonical images, despite their simplicity, fail to accommodate the complexity and randomness of motion, leading to artifacts in reconstructed images. 
conclusion: 
 In this paper, we present HeroMaker, an innovative human-aware framework that prioritizes human-centric video editing. Our approach utilizes motion priors based on human body mesh to establish the transformation correspondence from human-aware canonical fields to each video frame. Powered by our devised video representation, we maintain meaningful and structural canonical fields that facilitate the subsequent synthesis of temporal coherent and plausible results in response to diverse user interactions. Extensive experiments and visual results demonstrate the superior performance of our HeroMaker, while ablation studies confirm the effectiveness of our design.  
abstract: 
 Video generation and editing, particularly human-centric video editing, has seen a surge of interest in its potential to create immersive and dynamic content. A fundamental challenge is ensuring temporal coherence and visual harmony across frames, especially in handling large-scale human motion and maintaining consistency over long sequences. The previous methods, such as diffusion-based video editing, struggle with flickering and length limitations. In contrast, methods employing Video-2D representations grapple with accurately capturing complex structural relationships in largescale human motion. Simultaneously, some patterns on the human body appear intermittently throughout the video, posing a knotty problem in identifying visual correspondence. To address the above 
 
 Appendix:  


Decision: The paper focuses on human-centric video editing, aiming to create immersive and dynamic content by modifying the human performer within the input video. Their motion representation is novel (though inspired from  existing representations), however, it's significance is underwhelming. The quality of the final results could potentially be improved. A clear limitation is that this method only works with tight-fit clothing, simple poses, and uniform lighting conditions, limiting its applicability to in-the-wild videos. The choice of methods to compare with is not justified properly. Those methods are meant to work on a wide variety of examples. On the other hand, the proposed method is just  tuned for human-centric video. For improved comparisons, the authors can refer to a recent paper (Zhu et al. Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance). Please note that I am not holding this against the authors, as this was submitted post the ACM-MM deadline. 

I personally feel that the paper is a borderline work. However, considering all reviews and the author response, I am recommending for an accept. Please note that the authors must incorporate the reviewers' suggested changes into the final version of the paper.

Reviews: review: paper_summary: paper_summary: The paper targets human-centric video editing which aims to create immersive and dynamic content by modifying human performer within the input video. A significant challenge in this field is maintaining temporal and visual coherency, particularly with large-scale human motion. Prior works struggle with issues like flickering, length limitations, and accurately capturing complex structural relationships. 

To overcome these challenges, the authors introduce HeroMaker which utilizes body mesh-based human motion warping (responsible for rigid transformations) and neural deformation-based margin refinement (responsible for non-rigidities) to ensure semantic correctness and temporal consistency. By leveraging dual canonical fields and motion priors, HeroMaker achieves temporally coherent and visually plausible results. The authors present comprehensive experiments to show that HeroMaker outperforms existing methods in terms of temporal consistency, visual quality, and semantic coherence.
strength_weakness: strength_weakness: - The dual canonical that enables identifying more accurate correspondences is novel
- The idea of neural margin refinements is novel in video generation/editing task. It is interesting to adopt commonly used deformation modeling approach in 2D space for video reconstruction.
- The paper presents comprehensive experimental results and comparisons, signifying their superior performance
limitations: limitations: Minor comments:
- L488: presentation -> representation
- L506: in 2 -> Fig. 2
paper_summary: paper_summary: This paper addresses the challenges in human-centric video editing, focusing on ensuring temporal coherence and visual harmony across frames, particularly when handling large-scale human motion. The authors propose a novel framework, HeroMaker, to tackle these issues by manipulating a person's appearance within input videos to achieve consistent inter-frame results. HeroMaker leverages body mesh-based human motion warping and neural deformation-based margin refinement to ensure semantic correctness in the video reconstruction process. By manipulating dual canonical fields and combining them with motion priors, HeroMaker synthesizes temporally coherent and visually plausible videos.
strength_weakness: strength_weakness: ### 1.The paper addresses challenges in human-centric video editing with SMPL approach.
### 2. HeroMaker combines body mesh-based warping and neural deformation-based refinement to ensure temporal coherence and visual quality.
### 3. The user study validate the superior performance of HeroMaker over existing methods.
limitations: limitations: ### 1. Image Quality in Figures: The edited results presented in the figures appear somewhat blurred, particularly in terms of texture details. It is unclear whether this is due to issues with the mesh rendering or other factors. 

### 2. Comparison with Advanced Methods: The comparison methods primarily include earlier T2V models and NLA-based models, but lack comparisons with more advanced techniques such as Fatezero [1] and Tokenflow [2] (ICLR 2024). From a visual perspective, these newer methods seem to generate finer details and higher quality results. Including a comparison with these state-of-the-art methods would provide a more comprehensive evaluation of the proposed approach’s strengths and weaknesses.

[1] Qi, Chenyang, et al. "Fatezero: Fusing attentions for zero-shot text-based video editing." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Geyer, Michal, et al. "Tokenflow: Consistent diffusion features for consistent video editing." arXiv preprint arXiv:2307.10373 (2023).
paper_summary: paper_summary: The paper presents a human-centric framework for video editing, which incorporates human motion priors to establish dual canonical fields, effectively converting video editing tasks into more tractable image editing problems. It demonstrates success in achieving both temporal consistency and plausibility in edited outcomes for videos featuring single individuals and multiple people.
strength_weakness: strength_weakness: 1.Important Topic: human-centric video editing is indeed a topic of large importance in the community given the direction of the field.

2.Reasonable 3D Human Optimization Strategy: The paper introduces a fresh perspective by incorporating the optimization of 3D human reconstruction outcomes with a consideration for editing effectiveness and user-friendliness, thereby furnishing video editing tasks with more accurate and suitable human representations.

3.Innovative Deformation Handling: The work cleverly separates deformation into rigid motion and non-rigid margin, and applies a neural refinement technique to the human silhouette edges. This strategy closes the gap between simplified canonical representations and realistic image output, demonstrating an innovative approach to achieving natural-looking edits.
limitations: limitations: 1.Video Duration Limitations: The study focuses on videos of up to 200 frames, which may limit generalizability. Including results from longer videos would enhance the validity of temporal consistency claims.

2.Insufficient Differentiation Though the paper cites high computational demands and user unfriendliness in comparison to methods like DynVideo-E, it fails to provide HeroMaker's specific editing times within the Experiments section. Clearer performance comparisons, including timing, are necessary to fully assert the paper's advancements.

3.Static Scene Bias: The framework appears tailored to videos with fixed cameras and stationary subjects. Clarification on or demonstration of HeroMaker's capability to handle videos with moving cameras and dynamic subjects would expand its applicability.
paper_summary: paper_summary: In this paper, a human-centric video editing framework named HeroMaker is proposed, which leverages motion priors and dual canonical fields to achieve inter-frame consistent results. By leveraging the motion priors based on the SMPLX, HeroMaker establishes the transformation correspondence from the canonical field to each video frame. A supplemental video demo demonstrates that this approach surpasses existing SOTAs and receives temporally coherent and visually plausible results.
strength_weakness: strength_weakness: 1. This paper proposes a novel video representation based on motion priors and breaks it down into two components: known human motion warping and neural margin refinement, which could effectively convert the human-centric video editing problems into image editing problems
2. Benefitting from the introduced neural margin refinement, it is convenient for editing videos of clothed bodies.
limitations: limitations: 1. The author claims that by optimizing bias ΔT, this method could solve the video editing problem of clothed human. However, from the pipeline and experiments, clothes generally fit well. What I want to know is, is this method still effective for some loose fitting clothes, like dresses?
2.  The method is light incremental, both motion prior and canonical fields have been presented in previous methods. It is necessary for the author to clarify the innovation and contribution of the paper.
3. For the section related to video reconstruction, especially the part of Neural Margin Refinement and the definitions of 'points within/outside the transformation relationship', I think it is obscure and difficult to understand. Can the author polish this part in the revision?
4. Others:
  -. Line214, finetuning
  -. L216 works
  - L316 Inappropriate references
  -. Paper readability. Too many "however" appear in the section of related work, I suggest that the authors could utilize other similar words.