MagicFight: Personalized Martial Arts Combat Video Generation

Authors: magicanimate  animateanyone
champ  dreamoving
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 07:10:14

Abstract: Amid the surge in generic text-to-video generation, the field of personalized human video generation has witnessed notable advancements, primarily concentrated on single-person scenarios. However, to our knowledge, the domain of two-person interactions, particularly in the context of martial arts combat, remains uncharted. We identify a significant gap: existing models for single-person dancing generation prove insufficient for capturing the subtleties and complexities of two engaged fighters, resulting in challenges such as identity confusion, anomalous limbs, and action mismatches. To address this, we introduce a pioneering new task, Personalized Martial Arts Combat Video Generation. Our approach, MagicFight, is specifically crafted to overcome these hurdles. Given this pioneering task, we face a lack of appropriate datasets. Thus, we generate a bespoke dataset using the game physics engine Unity, meticulously crafting a multitude of 3D characters, martial arts moves, and scenes designed to represent the diversity of combat. MagicFight refines and adapts existing models and strategies to generate high-fidelity two-person combat videos that maintain individual identities and ensure seamless, coherent action sequences, thereby laying the groundwork for future innovations in the realm of interactive video content creation.

Main paper: 
 introduction: 
 Video generation has emerged as a prominent field in AI research in recent years, with the creation of personalized videos representing a subtask of significant commercial and artistic value. When the specified subject is a human, this process, also known as character animation generation, entails providing an image of the source character, whereupon the model generates a realistic video following a sequence of poses specified by the user. This task boasts many potential applications, including online retail, entertainment videos, art creation, and virtual characters, among others. Numerous studies have explored image animation and pose transfer by GAN [6, 34, 37-39, 50, 53, 56] , serving as foundational work.In recent years, diffusion models [14] have demonstrated their superiority in text-to-image [2, 19, 29, 33, 35, 36] and video generation [5, 9, 11, 13, 15, 16, 22, 31, 40, 43, 46] . Numerous researchers have utilized the architecture of diffusion models to explore video generation conditioned on given image [8, 11, 43, 55] . However, when applied to human animation, for which they are not specifically designed, these methods often produce character appearances that do not match the original image, leading to videos that lack movement coherence. For fashion video generation, DreamPose [21] introduces an adapter to fuse CLIP [32] image features into Stable Diffusion [35] and finetunes on the input sample.Recent works specializing in human dance video generation, including DisCo [42] , MagicAnimate [47] , AnimateAnyone [18] , MagicDance [7] , DreaMoving [10] and Champ [57] exhibit similar approaches and network structures. DisCo [42] extracts character and background features via ControlNet [52] while it shows serious flaws in generating the ID. Other methods [7, 10, 18, 47, 57] all aim to solve the issue of ID appearance. Each employs its own appearance encoder, utilizing a parameter-rich encoder like ControlNet for multi-scale and detailed ID feature extraction from the original image. They design an effective pose guide for controllability and a temporal module for smooth interframe transitions. Furthermore, the pivotal element is the training data they have amassed. By leveraging large-scale, high-quality datasets, these methods can animate arbitrary characters.However, all the aforementioned methods fall short in human fighting video generation involving multiple subjects. As these methods are designed for single-person dancing, they accept a single ID and a single-person pose sequence, and their training datasets predominantly contain single-person dance videos such as TikTok [20] . Besides, the absence of network design for multiperson and the lack of multi-person dataset preclude these existing works from effectively generating multi-person fighting videos. Hence, we introduce a new task: personalized martial arts combat video generation. There are three primary distinctions between our new task and the existing ones: 1) Subject number: The existing task focuses on solo dances, whereas ours involves two individuals. 2) Motion type: While fashion and dance videos emphasize slow and individual movements, martial arts combat requires capturing complex kung fu and varied poses. 3) Interaction dynamics: Unlike solo dance with no interactive dynamics, martial arts combat necessitates depicting the intricate interplay between two-person, highlighting the authenticity of the generated video.In this paper, we design a base method MagicFight for our proposed new task named personalized martial arts combat video generation. To establish this foundational method, we address existing issues in current techniques and investigate dataset production, processing, and training strategies. Our main contributions include:(1) For the first time, we delineate two-person fighting from one-person dancing. We create a dataset of martial arts combat videos named KungFu-Fiesta (KFF) and establish data cleaning rules for dataset quality and diversity, laying a solid foundation for this new task.(2) We introduce a multi-modal personalized network to learn conditioning on two reference IDs, pose, background, and prompt, focusing on the dynamic complexities of combat.With the personalized attention layer (ID-attn), we address the clothing and body misattribution problem in our task. (3) In the inference stage, we introduce body-shape adaptive strategy to automatically adjust the preset pose map, aligning the generated video more closely with the expected body shape. For arbitrary long video generation, we use a clip fusion technique to ensure continuity between clips. (4) We conduct a comprehensive ablation study from both the dataset and model training perspectives. We explore the properties, size, and quality requirements for our dataset on this task, and offer insights and guidance about the effect of different training components on overall performance. We creatively propose the Mixture Data Finetuning strategy, which mixes self-made two-person fashion videos and KFF dataset for training, in order to take full advantage of different data domain. 
related work 2.1 conditional video generation: 
 The field of video generation has advanced significantly, thanks to diffusion models adapted from text-to-image (T2I) techniques. Research efforts [9, 15, 16, 22, 26, 31, 40, 46, 49] introduce frame attention and embedding temporal layers within T2I models. Initiatives like Video LDM [5] advocate for image pretraining before engaging in video temporal training, and AnimateDiff [11] brings motion modules to T2I models without the need for specialized adaptation.Expanding into image-to-video transformation, VideoComposer [43] stands out by integrating images as conditional inputs during training. VideoCrafter [8] distinguishes itself by melding text and visual features from CLIP into its cross-attention mechanism. The Stable Video Diffusion (SVD) [4] signifies a quantum leap in enhancing video quality and dynamic representation. With W.A.L.T [12] pioneering through its VAE Encoder in choosing optimal latent representations, and the Sora [1] setting new standards for highdefinition, realistic video outputs, these advancements mark a decisive turn towards refined, high-quality video generation.Recent studies highlight the incorporation image-to-video diffusion model into human video generation. PIDM [3] introduces texture diffusion blocks to infuse desired texture patterns into the denoising process for human pose migration. LFDM [28] synthesizes optical flow sequences in latent space, distorting the input image based on specified conditions. LEO [44] represents motion through a series of flow maps, using a diffusion model to synthesize the motion sequence. DreamPose [21]  
methods: 
 First, we analyze the existing problems in Sec. 3.1. Then, we detail our dataset creation process in Sec. 3.2, and our model architecture in Sec. 3.3. We describe the training and inference in Sec. 3.4 and 3.5, respectively.We commence with an analysis of the challenges that existing models face in generating scenes with complex character interactions as shown in Fig. 1 . 1) During the generation with two-person interactions, a common issue is misattribution of clothing and body parts, particularly when characters are close. For instance, the woman's left leg in the short skirt might be incorrectly merged with the man's pants, with color inaccuracies also occurring. These issues highlight the necessity for a customed two-person model to address the misattribution problem. 2) Besides, missing body parts also frequently occur, like duplicated legs or absent feet. This issue stems from the inadequate data on leg lifting and kicking actions in the human dance dataset and the absence of foot keypoints in the pose maps, leading to the model's poor perception of leg and foot features, which is thirsty for a tailored martial arts dataset. 3) Moreover, existing models often produce medium-sized characters, overlooking the diversity in body shape. For instance, muscular "Hulk" and bony people are frequently underrepresented. Hence, we aim to solve the problem of mismatch between body type and given pose, adapting to any body shape during inference. Our research seeks to mitigate the aforementioned problems and lay the groundwork for future endeavors.This section details our first martial arts combat video dataset named KungFu-Fiesta (short for KFF). We make 4 scenarios for this dataset creation and finally chose the Unity scenario, details about it are in our appendix. With Unity, an advanced game physics engine, it can create highly realistic 3D character models and action animation in a simulated world, and by rendering the scene from an angle and exporting them to video, it is possible to create a large number of highly realistic martial arts combat videos. For the diversity and complexity of the dataset, we design hundreds of character IDs with different identities, covering more than 100 kinds of paired fighting actions, and a variety of shooting angles in 20 different scenes. After careful design and production, we capture more than 500 high-quality videos. Each video sample is about 10 seconds with 60 fps, ensuring the coherence of the action. In KungFu-Fiesta, each sample contains a combat video, two reference images of character IDs (for short reference IDs), and a pose map sequence, providing researchers with more conditions. The details of the dataset are shown in Table 1 .Our model is an extension of the Stable Diffusion (SD), so we inherit its VAE [23] , denoising UNet and CLIP encoder. Fig. 2 provides an overview of our framework. The input to the network is multiframe noisy latent 𝑧 𝑡 ∈ R 𝐹 ×𝑐 ×ℎ×𝑤 (timestep 𝑡). In order to utilize the general knowledge of human motion, our model is based on the pretrained model of AnimateAnyone [18] which is for singleperson dancing video generation. The framework consists of three key components: 1) ReferenceNet is responsible for encoding the appearance of the two reference IDs; 2) Pose Guider is for controlling the two-person's fight by pose map; 3) Temporal layer, the attention layer between these frames is to ensure the continuity of the character's movement. AnimateAnyone proposes to use a lightweight pose controller with only 4 simple convolutional layers since the pose control of single-person is easy. However, our twoperson martial arts situation is more complex and the input pose becomes two-person. We finally choose to use a large Pose Guider like ControlNet. Personalized Attention Layer. In our task, the given reference IDs provide detailed appearance information. However, the ReferenceNet of AnimateAnyone is designed and trained for singleperson feature extraction. Thus, we feed 2 ID images into Refer-enceNet alongside the batchsize dimension to extract their features [𝑟 1 , 𝑟 2 ], and then they are fed into the denoising U-Net. As shown in Fig. 2 , our personalized attention (ID-attn) layer replaces the original self-attention layer of SD. Given the feature map 𝑥 𝑙 ∈R 𝐹 ×ℎ×𝑤×𝑐 in the 𝑙-th ID-attn layer and ID features 𝑟 1 , 𝑟 2 ∈ R ℎ×𝑤×𝑐 , ID-attn is performed as: Ō𝑙,𝑖 =MaskAttn(𝑄 𝑙 , 𝐾 𝑖 , 𝑀 𝑖 )𝑉 𝑖 ,EQUATIONwhere Conditioned Background. For conditioned background (pure white also OK), we concat the given background image latent with 𝑧 𝑡 at channels and input to U-Net. The user can 1) provide a simple background image for end-to-end background customisation, and 2) if the user does not want to provide a background image, the conditioned background will be set to pure white, and then user can provide text prompt in Background Crafter to generate the background. Our Background Crafter is based on SDXL-Inpainting [30] . Its conditions are the original foreground image, background mask (it is easy to obtain a mask due to the white background), and text prompt. We finetune it on our dataset and follow [25] to maintain inter-frame background consistency, which is detailed in our appendix. made based on the UBC dataset [51] . It is worth noting that the UBC dataset originally contains videos of a single person walking down a fashion runway, and by splicing two randomly selected videos left and right together, we create a video dataset that simulates a two-person fashion runway, which has the advantages of a pure white and clean background, real people in the subjects, and highdefinition clothing textures, and the disadvantages that the two subjects are randomly spliced together, and lack of multi-subject interactions. Based on the benefits of the two-person fashion dataset, we hypothesise that a strategy of training with a mixture of twoperson fight videos and two-person fashion videos would improve the consistency and aesthetics of the appearance, thus demonstrating stronger generalisation capabilities when dealing with complex character interaction scenarios.𝑄Finetuning. Since our MagicFight model is finetuned on the pretrained Moore-AnimateAnyone [24]1 , we propose the 2 stages finetuning. The first stage during finetuning is a spatial learning stage, using individual video frames from our KFF dataset as image input. In denoising U-Net, all temporal layers are frozen and become intra-frame attention, and the model takes the noisy single frame as input, along with the reference IDs and the pose map. ReferenceNet and Pose Guider are trained to learn the spatial distribution of the two-person fighting. The pretrained weights on the human dancing dataset are used to initialize our denoising U-Net, ReferenceNet, and Pose Guider, which adopts a ControlNetlike structure rather than the lightweight controllers in Animate Anyone. The second stage is for temporal layer finetuning, whose input is 20 frames of video clip from our KFF dataset, and network parameters except temporal layers are frozen to learn the general law of two-person fighting action.technique to ensure continuity of details between clips. Specifically, we retain the 𝑥 𝑡 of the last 4 frames of each clip in sampling steps. When inferring the next video clip, we use the saved 4 frames and the following 20 frames as 𝑥 𝑡 . During each sampling step, we superimpose the 𝑥 𝑡 of the last 4 frames onto those of the first 4 frames of the currently generated clip to generate videos of arbitrary length maintaining consistency. 3.5.2 Body-Shape Adaptive Strategy. In the video generation process, considering the possible differences in body types (e.g., height, body shape, etc.) between the given pose map and the reference IDs, we face a challenge to ensure the body shape in the generated video is consistent with those in the reference IDs. For example, if the reference ID is a tall and chubby person, and the given pose is from a little girl, it may result in visual incongruity. Thus, we introduce a body-shape adaptive strategy, which is shown in Fig. 4 . First, we predict the pose of the reference IDs, and compute the center of mass of the character's keypoints in the horizontal (x-axis) and vertical (y-axis) directions. Similarly, we also compute those of the given conditioned pose map. Subsequently, we compute the body scale factors in the x/y-axis. With these scale factors, we scale the coordinates of all the key points in the pose to ensure that the generated video content meets the action requirements and is faithful to the body shape of the reference IDs. 
experiment 4.1 implementation: 
 To validate MagicFight's efficacy in generating martial arts combat videos with diverse IDs, we make two benchmarks, KFF reconstruction benchmark and open-set combat generation benchmark, to evaluate our model. We employ pretrained DWPose to estimate pose maps, including body, hands, and foot. All finetuning experiments are conducted on 8 NVIDIA A6000 GPUs, each with 48G GPU memory. In the first finetuning stage, we sample individual frames at a frame interval of 6, then adjust the frames to a resolution of 704×512. Finetuning is performed for 20,000 steps, with a batchsize of 2 per GPU. In the second finetuning stage, we finetune the temporal layer for 10,000 steps with a video sequence of 20 frames, frame interval of 6, and batchsize of 2. Both learning rates are set to 2e-6. During inference, we employ the DDIM sampling for 25 denoising steps. We adopt our long video generation technique and body-shape adaptive strategy for better generation. For comparison with human dance generation methods, we test all methods on the same benchmark, detailed in Sec. 4.2.Figs. 5 and 6 illustrate our method's capability to produce controllable combat videos for various character types, such as real, cartoon, robotic, and humanoid. Our method produces high-definition videos with realistic character details. It ensures temporal consistency with the reference IDs and maintains continuity between frames, despite significant motion.To illustrate our method's superiority over other video generation methods, we assess them on two bespoke benchmarks: KFF reconstruction benchmark and open-set combat generation benchmark. For quantitative evaluation of the reconstructed video quality, we utilize metrics such as SSIM [45] , PSNR [17] , FVD [41] , and LPIPS [54] . The evaluation of the open-set video generation benchmark incorporates user ratings, FVD [41] , and NIQE metrics [27] . In our experiments, we follow the computation of FVD as VideoGPT [48] .Given that SSIM and PSNR may not match human perception, we employ LPIPS and NIQE as complementary evaluation metrics. 2 , where our results significantly surpass those of other methods, particularly in the reconstruction metrics. Qualitative comparisons are displayed in Fig. 5 . We employ the web demo or the code of the compared methods. These methods can't provide conditional backgrounds or generate new backgrounds, and DreaMoving is a vertical screen resolution (so we keep it vertical). Refining human details demands high precision, while our method maintains detail consistency.Benchmark. The open-set combat generation benchmark focuses on the open world of human interactions video. We collect 20 IDs from the game community and the Internet, comprising 40 test samples. We generate about 10 seconds of video for each sample. The selection criteria for this benchmark allow characters, actions, and backgrounds to span any data domain, with no restrictions on data source. Our approach yields the best quantitative results, as shown in Table 3 . DisCo, An-imateAnyone, and MagicAnimate undergo extensive pre-training on human image datasets, learning basic single-person patterns, thus lacking multi-person interaction knowledge. In contrast, our mixture dataset training on the KFF and two-person fashion dataset yields superior results compared to these methods. Our method demonstrates that without explicit segmentation, the model can discern foreground-background relationships from multi-subject movements. Furthermore, our model excels at maintaining visual continuity in complex action sequences, demonstrating robustness in handling varied character appearances.To elucidate the differences in dataset attributes and explore their impact on finetuning efficacy, we focus on the data scale, number of character IDs, actions, backgrounds, and the mixture with two-person fashion data. Data Scale. Data scale is a key factor in evaluating the finetuning effectiveness. Theoretically, a larger dataset is believed to provide richer information for training, enhancing the model's generalization to new scenarios. Table 4 indicates that as the data scale increases, the model shows improvement in FVD and user scores exhibiting superior visual quality.Number of Character IDs. Among the attributes, the number of character IDs is a crucial factor under the assumption that more IDs offer diverse learning opportunities for character traits, thereby enhancing video diversity and realism. As depicted in Fig. 7 , 4 demonstrate our dataset's superiority across attributes. These results validate our hypothesis highlighting prioritizing character diversity in dataset construction. Number of Action. We hypothesize more actions in our dataset should help the dynamics and complexity of martial arts videos. The result presented in Table 4 and Fig. 7 shows that an increase in action types somewhat improves video quality metrics, though not as significantly as with IDs, suggesting that ID diversity is more crucial than action variety. Qualitative analysis reveals that more actions yield videos with complex interactions like overkick.Therefore, for open-set actions, the dataset should be constructed to include as many diverse martial arts types as possible.Mixture Dataset Finetuning. We explored the impact of using the mixture video dataset. Specifically Finetuning Module Ablation. Module-specific finetuning targets for the optimization of specific parameters while retaining the most original generative capabilities. We hypothesize that finetuning different modules has different effects. Table 4 and Fig. 7 present results of differences in finetuning modules, leading us to the following preliminary conclusions: 1) Without finetuning the denoising U-Net, denoising loss can only be reduced to around 0.4 but not further to 0.2. 2) Untrained ReferenceNet or Pose Guider leads to body distortions, missing parts, or inconsistent IDs. 3) Although the first stage of finetuning may yield suboptimal results, performance can be significantly improved in the second stage. 4) Finetuning solely the temporal layers often causes artifacts, distorted body, and background anomalies in certain samples.Incremental Data Finetuning. We initially finetune with mediumsized data and, after every 10,000 steps, gradually introduce new data. The results reveal that its impact on enhancing diversity and realism is negative. We hypothesize that gradually increasing the data scale may lead to a suboptimal model weight. 
limitations and future directions: 
 This part discusses the limitations of our proposed methodology and outlines directions for future research. Our approach has the following limitations: Firstly, like many visual generative models, ours struggles with perfect foot and hand generation. Secondly, our reference IDs offer only a single-angle view, making the generation of occluded parts during action problematic; for instance, if the reference image lacks a frontal view, the generated facial quality is poor. Thirdly, when the two people overlap for some complex action like wrestling, pose control becomes chaotic.Then we introduce our future work. First, when handling complex dresses, like cartoon costumes or clothes with ribbons, our method may exacerbate flash frame issues. We suggest manually labeling the pose map. Secondly, background control remains a significant challenge. The existing framework cannot generate backgrounds that are dynamic (such as flowing water, fire, and rain), have complex layouts, or have passers-by. We are working hard to propose a new framework that can generate dynamic foreground and background in the same model. Finally, our current approach focuses on the case where the camera is stationary, and all of our training videos are camera-still. In order to adapt to the situation of dynamic shots in real martial arts movies (e.g., complex situations such as slow camera movement, rotation, or even switching of shots, etc.), future work will focus on introducing modeling of the camera position for the network and producing more video datasets with camera movement, which will lead to a more realistic and higher-degree-of-freedom generation of martial arts videos. 
conclusion: 
 This paper introduces a foundational framework for generating martial arts combat videos, transforming two characters into combat video with pose sequences, ensuring appearance consistency and temporal stability. We make the first combat video dataset named KungFu-Fiesta (KFF), specifically designed for this task, created using the Unity engine to ensure diversity and physical realism. We finetune a multi-modal personalized network to acquire combat knowledge, aiming to preserve the intricate appearance of IDs while enabling efficient pose control and temporal continuity. The user can specify a background image or easily customize the background through the Background Crafter by text prompt.  
 
 Appendix: Since AnimateAnyone has no released code, we use reproduction version. 


Decision: The paper introduces "MagicFight," a framework designed for generating text-to-video content focusing on two-person martial arts interactions. This approach incorporates a specially designed dataset to address the unique challenges of accurately depicting complex interactions in martial arts combat. Experimental results support the effectiveness of MagicFight.

Strengths:

+ Relevance and Innovation: The focus on two-person interactions in video generation, specifically in martial arts, fills a gap in the current research landscape, which has predominantly concentrated on single-agent scenarios.

+ Structured Presentation: The paper is well-organized, making it accessible and easy to comprehend, which is beneficial for both seasoned researchers and those new to the field.

+ Effective Methodology: The proposed method not only addresses the generation of complex video content but does so with a level of performance that surpasses existing methods, as demonstrated through both qualitative and quantitative assessments.

Limitations:

+ Scope of Method Application: While the method excels in two-person scenarios, the paper lacks a detailed discussion on its applicability or adaptation to scenarios involving more than two individuals or less controlled environments.

+ Detail on Specific Challenges: The paper does not thoroughly explain how it resolves specific technical challenges unique to two-person interactions, such as identity preservation and interaction dynamics, which are critical for realistic video generation.

+ Availability of Demonstrative Materials: There is no direct provision of links or supplementary materials that allow for the independent evaluation of the generated videos, which could help in better assessing the visual quality and effectiveness of the proposed method.

The recommendation leans towards a borderline acceptance, indicating potential for impact but also highlighting areas where further detail and clarification could strengthen the overall contribution. The confidence in this recommendation is somewhat confident, reflecting the need for additional material to fully assess the method's robustness and generalizability.  It is highly recommended that the code be made publicly available.

Reviews: review: paper_summary: paper_summary: The paper focuses on text-to-video generation with two-person interactions. Thus, the paper proposes MagicFight and a bespoke dataset for conditioned video generation in the context of martial art combat. Experimental results showed the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The topic, i.e. multi-agent conditional video generation is more general with more applications, while remains less unexplored in the literature.
2. The proposed method seems reasonable and can achieve good performance.
3. The paper is well structured and easy to follow.
limitations: limitations: 1. Although two-person fighting cases are not considered by previous methods, there already exist methods for multi-person conditional video generation, which is a more general task. What are the specific challenges of two-person fighting video generation? What specific designs are proposed to tackle the challenges? 
2. How to alleviate identity confusion, model complex interactions between identities, and ensure consistency over time are the main challenges in multi-agent conditional video generation. Although the authors claim that the proposed method is specially crafted to overcome these burdens, how and why the method can alleviate these are not clearly stated.
3. Are there any anonymous links that can show the generated videos by the proposed method?
paper_summary: paper_summary: This work introduces a new method for generating personalized martial arts combat videos, addressing the complexities and challenges inherent in two-person interactions. Unlike existing models that focus on single-person scenarios such as dancing, MagicFight aims to generate high-fidelity combat videos that maintain individual identities and coherent action sequences. To support this, the authors created a bespoke dataset using the Unity game engine, ensuring a diverse range of combat scenarios.
strength_weakness: strength_weakness: -  This work introduces MagicFight, a new task of personalized martial arts combat video generation, filling a significant gap in existing research that primarily focuses on single-person scenarios like dancing.

-  The KungFu-Fiesta (KFF) dataset, created using Unity, provides a rich and diverse collection of combat scenarios, enabling the model to learn from a variety of interactions and poses.

-  The paper includes thorough qualitative and quantitative evaluations, demonstrating the superiority of MagicFight over existing methods.
limitations: limitations: -  Given two reference IDs and the pose sequence, the proposed framework is able to generate a suitable video. Usually, how can we obtain the pose sequence for a video, especially in real-world scenarios rather than synthetic scenarios? How can we align the pose sequence with the reference IDs and is there a possibility that the pose sequence could be inconsistent with the described language or reference IDs?

-  Compared to generating a single person, generating interactions between two people focuses on the difficulty of creating decoupled actions due to potential mutual occlusions. I guess that the model could struggle with generating occluded parts during action sequences, especially when two characters overlap, which affects the realism and accuracy of the generated videos. 

-  The integration of various modules and the use of a bespoke dataset might lead to increased computational complexity and demand, potentially limiting scalability. The authors could also show the count of parameters and inference time in the experimental section. 

-  Some references related to action generations are missing [1,2,3], which can be added. 

[1] Tm2d: Bimodality Driven 3d Dance Generation via Music-text Integration, in ICCV2023. 

[2] Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model, in CVPR2024.

[3] StructLDM: Structured Latent Diffusion for 3D Human Generation, in arxiv2024.
paper_summary: paper_summary: The paper introduces MagicFight, a novel framework for generating personalized martial arts combat videos, filling a gap in the field of two-person interaction video generation. Specifically, MagicFight learns from two reference IDs, pose sequences, and other conditions, employing a personalized attention layer to address misattribution issues in character generation.

Additionally, it presents the creation of KungFu-Fiesta (KFF), a bespoke dataset designed using Unity to capture the nuances of martial arts combat with a diverse range of characters and scenes, crucial for training the model.

MagicFight incorporates a body-shape adaptive strategy for inference, aligning generated videos with expected body shapes, and a clip fusion technique for generating long, coherent videos.

The paper provides a comprehensive evaluation of MagicFight, demonstrating its effectiveness in producing high-fidelity, temporally consistent combat videos and its superiority over existing methods through both qualitative and quantitative benchmarks.
strength_weakness: strength_weakness: 1. The problem is interesting and important, i.e., the domain of two-person interactions, particularly in the context of martial arts combat, remains uncharted.

2. The motivation is clear and the proposed framework is novel. 

3. Extensive experiments have been conducted. MagicFight achieves robust performance.

4. The proposed dataset will make a significant contribution to the video generation community.
limitations: limitations: Lack of User Study: Conducting a more extensive user study to gather feedback on the generated videos' realism, diversity, and user satisfaction could provide valuable insights for further refinement.
paper_summary: paper_summary: This paper proposes a controllable multi-person human-centric video generation method, MagicFight. The major contribution in this paper is a personalized attention module, which can control the appearance of different person based on the ID. Furthermore, this paper also propose a new dataset on martial art that targeted for multi-person human-centric video generation. This paper achieves strong performance on several benchmarks.
strength_weakness: strength_weakness: 1. Multi-person video generation can be challenging, and this paper proposes an effective method towards this task.
2. New multi-person video dataset is proposed in this paper, which could potentially serve as benchmarks for future work.
3. The quantitative results are good, outperforming existing methods.
limitations: limitations: 1. This paper uses mixed dataset finetuning (Sec 3.4), which could pose an unfair comparison with competing methods.
2. This paper is not the first one to work on multi-person generation. For example, [a] is an zero-shot methods, and this paper does not compare the results with [a], but only compare results with single-person generations methods (MagicAnimate, AnimateAnyone, Champ, and DreaMoving), which is unfair.
3. This paper works on **video** generation, but does not provide video comparison. Video metrics are well-known for the fluctuation; thus, only from the metrics, the visual quality and temporal smoothness cannot be well evaluated.
4. Generalization ability is not evaluated. This paper only conducts in-domain evaluation and misses out-of-distribution evaluations, such as using non-human reference images.
5. This paper only shows the results of two persons. I wonder if this method can generalize to more people?



Missing references

[a] Xu, Zhe, Kun Wei, Xu Yang, and Cheng Deng. "Do You Guys Want to Dance: Zero-Shot Compositional Human Dance Generation with Multiple Persons." *arXiv preprint arXiv:2401.13363* (2024).



Justification: my major concern lies in the incomplete experiments and lack of video result comparison.
paper_summary: paper_summary: This article introduces a new task: personalized martial arts combat video generation. It proposes a specialized dataset constructed using Unity to support this task, which includes 3D characters, actions, and scenes.
strength_weakness: strength_weakness: 1. This paper introduces a new task, which is interesting. The combat videos, of course, are more complex than single dance videos.
2. A new dataset should be considered as a contribution. Especially when it is a new task.
3. This work did consider some detail problems, such as body size adaptive strategy.
limitations: limitations: 1. My greatest concern is about the authenticity of the results presented in the paper. I have observed that the method proposed in the paper shows abnormal rendering of light and shadow in some results, particularly in parts that appear to be 3D assets. This is especially evident in the example of the two girls fighting in the upper part of Figure 5. It is clear that the edge shadows of the input images are significant and are reflected in other methods. However, the results of the proposed method exhibit an unusually smooth and markedly different skin shading. 
2. The method presented in this paper is also difficult to comprehend. I noticed that the inputs to the method are character IDs, text descriptions, background images, and pose sequences. However, additional character mask information appears in the Personalized Attention proposed in the paper. Moreover, it is determined before the VAE whether the background is generated through text. Firstly, how is this mask obtained? Can masks for different characters be estimated based on the given pose sequences for various reference IDs? Secondly, the text-generated background is confusing, with the example "two men fighting in an abandoned factory." The only useful information in this sentence is the abandoned factory. Furthermore, for the given method, it seems that it cannot be any other choice besides two men, nor any other action besides fighting, even though these are essentially useless information. Lastly, I believe the most useful part of the method is the character ID and pose. Is the text description truly meaningful?
3. The proposed task is met with skepticism in its potential applications. The task presented in the paper requires the synthesis of fighting videos using reference IDs and pose sequences. However, the information from the reference IDs is limited, as mentioned in the limitations, which can lead to issues with occluded parts. The information loss due to the modality is irremediable. Moreover, the action sequence is the most significant workload in the given application. I believe that the task of generating action sequences is more valuable than the task proposed in the paper, as the existing process can quickly render videos from action sequences and 3D assets, including more complete and detailed results, scenes, and lighting. In contrast, the task proposed in this paper does not reduce the input requirements (both need pose and character, and additionally require text) or improve the quality of the results (due to the modal disadvantage of 2D characters and the significantly inferior lighting inference capabilities compared to existing renderers).
4.  In summary, I currently have a high degree of suspicion regarding the authenticity of the results. Moreover, I believe the method is overly convoluted, making it difficult for me to capture the genuinely useful aspects. Additionally, while the new task is intuitively interesting, I think it is less effective in the application scenarios mentioned in the paper compared to other technological approaches.