HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness

Authors: zihui  xue
mi  luo
changan  chen
kristen  grauman
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 04:04:35

Abstract: We study the problem of precisely swapping objects in videos, with a focus on those interacted with by hands, given one user-provided reference object image. Despite the great advancements that diffusion models have made in video editing recently, these models often fall short in handling the intricacies of hand-object interactions (HOI), failing to produce realistic edits---especially when object swapping results in object shape or functionality changes. To bridge this gap, we present HOI-Swap, a novel diffusion-based video editing framework trained in a self-supervised manner.  Designed in two stages, the first stage focuses on object swapping in a single frame with HOI awareness; the model learns to adjust the interaction patterns, such as the hand grasp, based on changes in the object's properties. The second stage extends the single-frame edit across the entire sequence; we achieve controllable motion alignment with the original video by: (1) warping a new sequence from the stage-I edited frame based on sampled motion points and (2) conditioning video generation on the warped sequence. Comprehensive qualitative and quantitative evaluations demonstrate that HOI-Swap significantly outperforms existing methods, delivering high-quality video edits with realistic HOIs.

Main paper: 
 introduction: 
 Consider a video depicting the process of a human hand picking up a kettle and moving it around (Figure 1 ). What if we want to replace the hand-interacting object in this scene with another item specified by the user-perhaps a differently-shaped kettle, a bottle, or a bowl? This capability-to swap the in-contact object in a video with another, while aligning with the original video's content-is crucial for enhancing various real-world applications. Such functionality can transform entertainment, allowing users to create novel video content without the need to re-record or engage in labor-intensive manual editing. For example, in advertising, there may be situations where a pre-recorded video needs to adapt to new sponsorship requirements by replacing a soda can in the video with a water bottle. Additionally, it holds significant promise in robotics, where recent results suggest generative models can reduce the reliance on manually collected task-specific visual data and thereby enable large-scale robot learning [25, 58] . For example, imagine a scenario where, from just a single video of a mug being picked up, a generative model is able to produce numerous variants of this video with diverse objects such as bottles, bowls and kettles. This capability could greatly streamline the data collection process, reducing the need for extensive manual data collection. However, hands are notoriously challenging to work with in image/video editing [62, 66] . They pose significant hurdles in manual photoshopping and often produce unsatisfactory outputs when automated by generative models. The precise swapping of hand-interacting objects presents a unique challenge that existing diffusion models [19, 45, 1, 56] , despite their advances in video editing, fail to address adequately. This difficulty arises from three main factors: the need for (a) HOI-aware capabilities, (b) spatial alignment with the source context, and (c) controllable temporal alignment with the source video's motion patterns. Figure 1 : We present HOI-Swap that seamlessly swaps the in-contact object in videos using a reference object image, producing precise video edits with natural hand-object interactions (HOI).Notice how the generated hand needs to adjust to the shapes of the swapped-in objects (A,B,C) and how the reference object may require automatic re-posing to fit the video context (A).First, consider the challenge of HOI awareness. Objects chosen for swapping often vary in their properties from the original, resulting in changes to interaction patterns. For example, as illustrated in Figure 2 (a), replacing the kettle from the original video with a bowl necessitates adjustments in the hand's grasp patterns. While many generative inpainting methods [57, 50, 36, 7, 51, 16] have been developed to insert reference objects into specific scene regions, they are generally limited to contexts where the objects are isolated-not in contact with a human hand or other objects-and thus lack HOI awareness. In Figure 2 (a), the two image inpainting approaches Paint by Example (PBE) [57] and AnyDoor [7] either merely replicate the hand pose from the original image, or produce unnaturally occluded hands, resulting in suboptimal and unrealistic HOI patterns.Second, consider the challenge of spatial alignment with the original video. The reference object might appear in any arbitrary pose; for instance, in Figure 2 (b), the kettle handle is on the left in the reference image, but for realistic interaction, the generated content needs to reposition the kettle handle to the right, where the hand is poised to grasp it. However, current approaches do not offer this level of control, as evidenced by the results from a hand insertion approach Affordance Diffusion (Afford Diff) [62] in Figure 2 (b). Despite being adept at generating a hand interacting with the given kettle, it does not ensure correct object placement to align with the hand and scene context in the original image, lacking spatial alignment capability.Third, consider the challenge of temporal alignment with the original video. We highlight a crucial observation in this problem: the motion information in an HOI video sequence is closely tied to the object's characteristics (such as its shape and function). This means that when swapping objects, not all motions from the source video are appropriate or transferable to the new object. For example, Figure 2 (c) shows a hand closing a trash can, alongside an HOI-Swap edited image where the original can is replaced with one that is differently shaped and functions differently. Ideally, the generated content should reflect the motion of closing the lid, yet it may not replicate the exact motion from the source video due to these differences. Conversely, Figure 1 (first row) depicts a scenario of swapping one kettle with another. Here, the objects undergo only slight shape changes, allowing the generated video to closely follow the source video's motion. While there are varying degrees to which object changes can affect the original motion, current video editing approaches [42, 12, 10, 59, 21, 6, 5, 34] adhere to a rigid degree of motion alignment, often targeting 100%, adopting conditional signals like optical flow or depth sequences that encode substantial object information. Consequently, they lack the controllability to adjust the degree of motion alignment based on object changes.To address these challenges, we introduce HOI-Swap, a video editing framework designed for precise object edits with HOI awareness. We approach the challenge by posing it as a video inpainting task, and propose a fully self-supervised training framework. Our innovative approach structures the editing process into two stages. The first stage addresses HOI awareness and establishes spatial alignment, by training an image inpainting diffusion model for object swapping in one frame. The second stage propagates this one-frame edit across the remaining frames to achieve temporal alignment with the source. For this, we propose to warp a video sequence from the edited frame using randomly sampled points tracked across frames with optical flow, and train a video diffusion model that learns to fill in AnyDoor [7] , and Affordance Diffusion (Afford Diff) [62] .the sparse and incomplete warped sequence. Our approach thus enables controllable motion alignment by varying the sparsity of sampled points. During inference, users can modulate this number based on the object's changes-sampling fewer or no points for significant shape or functional alterations, and more points for minor changes to closely replicate the source video's motion. HOI-Swap is evaluated on both image and video editing tasks, consistently producing high-quality edits with realistic HOIs. It greatly surpasses existing editing approaches in both qualitative and quantitative evaluations, including a user study. By extending the capabilities of generative models into the largely unexplored domain of HOI, we hope that HOI-Swap opens new avenues for research and practical applications in this innovative field. 
related work: 
 Generative Models for Image Editing Recent advances in diffusion models [19, 45] have significantly enhanced the capabilities of image editing. Predominant models [38, 18, 2, 26, 53, 24 ] use text as guidance, which, despite its utility, often lacks the precision needed for exact control. In response, a growing body of studies have begun to explore the use of reference images as editing guidance. Customized approaches like Textual Inversion [11] and DreamBooth [47] are designed to generate new images of a specific object given several of its images and a relevant text prompt. However, these methods require extensive finetuning for each case and lack the ability to integrate the object into another user-specified scene image. More closely related to our task, a few approaches aim to seamlessly blend a reference object [57, 50, 36, 7, 51, 16] or person [29] into a specific region of a target scene image. However, as demonstrated in Figure 2 and detailed in Section 4, these methods prove inadequate, often producing images with unnatural HOIs, such as missing fingers, distorted hands, or oddly shaped objects. These issues reveal shortcomings in current generative models, and motivate the development of HOI-Swap.Generative Models for Video Editing With the advent in diffusion-based text-to-image and textto-video generation [45, 1, 13] , many efforts explore extending pretrained diffusion models for video editing, employing zero-shot [12, 59, 42, 4, 60, 28] , or one-shot-tuned learning paradigms [55, 32, 17, 67] . However, these methods require extensive video decomposition or costly per-video fine-tuning, with processing times ranging from several minutes to multiple hours on high-end GPUs for a single edit, which curtails their usability in practical creative tools. Another line of work [10, 6, 34, 54, 63, 41] adopts a training-based approach, where models are trained on large-scale datasets to enable their use as immediately effective editing tools during inference. Our work falls into this training-based paradigm.Similar to image editing, the majority of video editing approaches [1, 13, 12, 59, 42, 4, 60, 55, 32] rely on text as editing guidance. Since text prompts may not accurately capture the user's intent [67, 14, 41] , for our task, using an object image provides more precise guidance. In terms of motion guidance, most video editing approaches [42, 12, 10, 59, 21, 6, 5, 34, 63] utilize per-frame structural signals extracted from the source video, including depth map, optical flow, sketches, or canny edge sequences, facilitated by well-trained spatial-control image diffusion models such as ControlNet [65] , T2I-Adapter [39] and Composer [22] . Since these structural signals inherently encode the shape of the original object, they are unsuitable for edits involving shape changes [41, 17] . Recent works explore shape-aware video editing, by the use of Layered Neural Atlas [32] , modifying the source video's optical flow sequences [41] , or establishing correspondence with sparse semantic points [17] . However, all these approaches overlook the impact of object changes on motion and only enforce a fixed degree of motion alignment. In contrast, HOI-Swap introduces flexibility by allowing users to adjust the sparsity of sampled points during inference, providing precise motion control.There is growing interest in generating plausible HOIs [20, 31, 62, 61, 9, 66, 64] . Affordance Diffusion [62] inserts a synthesized hand given a single object image. Similarly, HOIDiffusion [66] aims to create HOI images conditioned on a 3D object model along with detailed text descriptions. Meanwhile, other approaches in the 3D domain focus on generating realistic 3D HOI interactions from textual descriptions [9] , synthesizing grasping motions [64] or reconstructing 3D HOIs from real videos [61] . Despite these advancements, the task of accurately swapping in-contact objects in videos remains unexplored. Our work directly addresses this gap.The realism of the hand-object interaction and how well the edited image aligns with the reference object's semantics (e.g., hand grasping varies for a mug and a toy car) and the original frame's scene and hand context.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We describe limitations of HOI-Swap in Supp. Section D and provide its failure cases in Supp. video. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We first formulate the problem and provide an overview of the two-stage HOI-Swap in Section 3.1. Section 3.2 and Section 3.3 detail the first and second stage of HOI-Swap, respectively.Given a source video V = {I i } N i=1 consisting of N frames (where I i represents the i-th frame), a binary mask (bounding box) sequence {M i } N i=1 that identifies the region of the source object to be replaced in V, and an image of the reference object I ref , the objective is to generate a modified video V ⋆ , that seamlessly swaps the original object in V with I ref .We create a fully self-supervised training approach, necessitated by the impracticality of collecting paired videos (V, V ⋆ ). We pose the problem as a video inpainting task. Specifically, from the original training video V and mask M, we derive a masked video V m , accompanied by a set of reference object images {I ref }. During training, the model takes the masked video sequence V m and an image I ref of the same object originating from a different random timepoint (and hence varying pose and/or viewpoint) in the same training video to reconstruct V. During inference, the model is given a bounding box-masked sequence combined with various object images to test its swapping capability.As outlined in Section 1, the task presents three main challenges: HOI awareness, spatial and temporal alignment. In response, we propose a two-stage framework that decomposes the complexity. The first stage integrates the reference object into a single frame, targeting HOI awareness and establishing spatial alignment initially. The second stage propagates the one-frame edit across the entire video sequence, focusing on temporal alignment and carrying forward the spatial alignment from the first stage. This structured approach effectively lightens the model's generation load. In terms of which frame to select from the input video for stage-I edits, we automate the process by adopting an off-the-shelf hand-object detector [49] to identify frames with hand-object contact, from which we randomly select one for editing. See Supp. C.4 for a detailed analysis. For greater applicability and flexibility, each stage is trained separately in a self-supervised manner, using the original object image as I ref . Consequently, the stage-I model not only plays a crucial role in the video editing pipeline but also serves as a standalone image editing tool. The full pipeline is illustrated in Figure 3 .In the first stage, given a source frame I i , a binary mask M i (where 1 indicates the object's region and 0 denotes the background) and a reference object image I ref , the objective is to generate an edited frame I ⋆ i where the reference object is naturally blended into the source frame. We aim for the generated content I ⋆ i to exhibit HOI awareness and spatial alignment with I i , ensuring that the In stage II, one frame is selected from the video to serve as the anchor. The remaining video is then warped using this anchor frame, several points sampled within it, and optical flow extracted from the video. A video diffusion model ϵ θ2 is trained to reconstruct the full video sequence from the warped sequence. During inference, the stage-I model swaps the object in one frame. This edited frame then serves as the anchor for warping a new video sequence, which is subsequently taken as input for the stage-II model to generate the complete video.reference object is realistically interacting with human hands and accurately positioned within the scene context of the source frame.Training is conducted in a self-supervised manner. Below, we detail how we prepare pseudo data pairs for training.• Masked frame preparation: For each frame I i , we obtain the object's bounding box (denoted by M bbox i ) from its segmentation mask M i . To prevent the bounding box shape from influencing the aspect ratio of the generated results, we adjust the bounding box to be square. The masked frame is thus derived by applying the square bounding box mask to the frame, yielding I m i = I i ⊙ M bbox i . We then crop and center the original frame to focus on the object region, enhancing its visibility.• Reference object preparation: We extract the object image I obj i = I i ⊙ (1 -M i ), which may be incomplete or partially obscured due to contact with hands or other objects. To address these occlusions, we employ an off-the-shelf text-guided inpainting model [40] to fill in the missing parts of the object, using the object's name as the text prompt.• Object augmentation: Directly forming a training pair (I obj i , I m i → I i ) is suboptimal as it does not reflect the variability expected in real-world applications. At test time, the reference object may appear in any pose, orientation, or size. To bridge this gap between training and testing scenarios, we apply strong augmentation techniques: (1) Spatially, we enhance the object's diversity by applying random resizing, flips, rotations, and perspective transformations; (2) Temporally, instead of using the object image from its original frame i, we introduce variability by randomly selecting an alternate frame, I obj i ′ , from all available frames in the source video that contain the object. Finally, for preservation of the reference object's identity and structural details, we collage the augmented reference object image I obj i ′ onto the masked region in I m i , producing I m ′ i . Model design We design the stage-I model as an image-conditioned latent diffusion model (LDM). Specifically, we employ a pretrained variational autoencoder (VAE) [27] E as used in prior work [45] to encode a frame I into a latent space representation. This encoding is denoted by z = E(I), where I ∈ R 3×H×W , z ∈ R 4×H/8×W/8 , H and W denote the frame's height and width, respectively. During the forward diffusion process, Gaussian noise is gradually added to z over T steps, producing a sequence of noisy samples {z 0 , • • • , z t , • • • , z T }. We train a denoising network ϵ θ1 (•, t) that learns to reverse the diffusion process, which is implemented as a UNet [46] . The training objective of stage I is summarized as:EQUATION)Our LDM is designed to take in two types of conditional signals alongside the standard z t and t: (1) The reference object imageI obj i ′∈ R 3×H×W is encoded through DINO [3] for distinctive object features d obj ∈ R 768 , d obj is then taken by ϵ θ1 as input to guide the denoising process via cross-attention mechanisms; (2) The masked frame I m ∈ R 3×H×W is encoded by the same VAE to produce z m = E(I m ). z m ∈ R 4×H/8×H/8 is then concatenated channel-wise with z before being fed into ϵ θ1 .In stage II, given the first-stage edit I ⋆ i , source video V, and its binary mask sequence M, the objective is to generate a new video V ⋆ that propagates the single-frame edit across the remaining frames. For this purpose, we perform warping to transfer pixel points from the edited frame to the remaining frames, resulting in a sparse and incomplete video sequence. We then train a video diffusion network that learns to correct and fill in the gaps, completing the video editing process.Training continues in a self-supervised manner. For this purpose, we use the original frame I i from V during training and only replace I i with the stage-I edit I ⋆ i during inference time. The detailed process is explained below:• Masked frame sequence preparation. We employ the same frame masking strategy in stage I and apply it to a sequence of frames V = {I i } N i=1 . To standardize the masking across the sequence, we identify the largest bounding box M bbox max from the sequence {M bbox i } N i=1 and mask each frame i byI m i = I i ⊙ M bbox max , resulting in a masked frame sequence {I m i } N i=1 .This ensures that the model is trained to inpaint a consistent object region across different frames.• Conditioning frame selection. To avoid limiting the model to generating videos based on a specific reference frame, such as the first frame as often used in existing image-to-video generative models, we randomly select a frame I c from V as the conditioning frame. This any-frame-conditioning mechanism brings additional flexibility during inference (for detailed discussion, see Supp. C.4).Given that object swapping can result in changes to shape or functionality, our approach is designed to be flexible and adaptable, allowing for varying degrees of motion pattern encoding from the source video. The key idea is to control the level of motion agreement with the source via points sampled within the masked region. Tracking a large number of points over time captures extensive motion information from the source video, but it also reveals much about the source object's characteristics (e.g. shape). Conversely, using few points reduces the motion information and object characteristics carried over, but offers the model more freedom to generate plausible motion for the target. The latter scenario is particularly useful when only partial motion transfer is desired due to differences between objects, as exemplified in Figure 2 (c).To be specific, our approach involves the following steps: (1) uniformly sample r% pixel points within the original object's region in the conditioning frame I c (i.e., where M bbox max equals 1), ( 2) track these points using optical flow vectors computed by RAFT [52] , and (3) warp a new video sequence based on these tracked points, using I c as the anchor. The resulting warped sequence is denoted byV warp = {I warp i } N i=1 .The anchor frame I warp c is intact and serves as the basis for warping (i.e., I warp c = I c ). For the other frames (i ̸ = c), we overlay these points tracked from the conditioning frame c to frame i onto the masked region of I i (i.e., I m i ), creating a composite image I warp i . See Figure 3 (upper right) for an illustration of this process.We vary the sparsity level of sampled motion points, r from 0 to 100 during training, preparing the model for a range of scenarios-from no motion guidance to full motion information. During inference, we demonstrate that the model is capable of generating video sequences with varying levels of motion alignment to the original video (see Section 4.2 and Supp. C.4 for further discussion). Thus, our stage-II training goal is to take the warped yet incomplete video sequence (V warp ) as input, then learn to fill in the gaps and smooth over discontinuities to reconstruct the full video (V).We design the stage-II model as a video LDM. The LDM architecture and training objective closely follow those of stage I, with some adaptations for handling video sequences. We continue to use the VAE encoder E, but now apply it to a sequence of frames V ∈ R N ×3×H×W to obtain the latent feature z ∈ R N ×4×H/8×W/8 . We inflate the 2D UNet into a 3D UNet as the denoising network (denoted by ϵ θ2 (•, t)) for generating a frame sequence. Specifically, we insert additional temporal layers into the 2D UNet and interleave them with spatial layers, following [1, 10] . The stage-II training objective is shown below:EQUATIONOur video LDM involves two conditional signals in addition to the standard z t (a noised version of z) and diffusion time step t: (1) The warped video sequence V warp ∈ R N ×3×H×W , is processed similarly as V, encoded by E, producing a latent feature z w ∈ R N ×4×H/8×W/8 . z w is then concatenated channel-wise with z before being fed into the denoising network ϵ θ2 . (2) The selected frame I c , besides serving as the anchor in the warping process, is also encoded by CLIP [43] , yielding d c ∈ R 768 that provides high-level contextual guidance and is incorporated in ϵ θ2 via cross-attention. Additionally, please see Supp. C.4 for an ablation of comparing CLIP and DINO as the object encoder.The seamlessness of the object swap, considering:• Does the generated object maintain the identity of the reference object image?• How realistic is the hand-object interaction?• How well does the edited image align with the reference object's semantics and the original frame's scene and hand context?For each video editing question, participants were asked to select the best one or two edits based on the following two evaluation dimensions:1. Motion Alignment: The extent to which the edit matches the motion of the source video.2. Overall Quality: The seamlessness of the object swap, considering:• Does the hand-object interaction look realistic?• How well does the edited video align with the reference object's semantics and the original video's scene and hand context? • How well does the motion in the generated video align with the original video?Recognizing the inherent challenges of the video editing task, we included a sanity check sub-question asking users if "All edits are of very low quality; none successfully swap the object."Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Please refer to our project website.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 To comprehensively evaluate HOI-Swap, we consider both the image editing (accomplished by the stage-I model) and video editing (facilitated by the entire pipeline) task. We describe the experimental setup in Section 4.1 and present editing results along with an ablation analysis in Section 4.2.Datasets Our training leverages two large-scale egocentric datasets, HOI4D [35] and Ego-Exo4D [15] , which feature abundant HOIs, making them particularly suitable for exploring this problem. In total, the stage-I model is trained on 106.7K frames and the stage-II model uses 26.8K 2-second video clips for training. The evaluation set for image editing includes 1,250 source images, each paired with four reference object images, resulting in a total of 5,000 edited images. The evaluation set for video editing is composed of 25 source videos, also combined with four object images each, yielding 100 unique edited videos. Note that: (1) For image editing evaluation, we include both hand-present scenarios, which challenge the model's HOI-aware capabilities, and non-hand-present scenarios (20%) to evaluate its general object-swapping capability. (2) For video editing evaluation, in addition to HOI4D videos, we include challenging in-the-wild videos from EPIC-Kitchens [8] and third-person videos from TCN Pouring [48] -datasets not used in training-to assess the model's zero-shot generalization across diverse scenarios. See Supp. B.1 for complete dataset descriptions.Baselines For image editing, our stage-I model is compared with two strong image inpainting approaches, PBE [57] and AnyDoor [7] , along with a hand-insertion approach Afford Diff [62] in the HOI domain. For video editing, we consider the following baselines: (1) applying the best image editing baseline to each frame of the video; (2) integrating the best image editing baseline with AnyV2V [28] , a recent video editing approach that takes one edited frame from any black-box image editing model as conditional guidance; (3) VideoSwap [17] , the state-of-the-art approach in customized object swapping. Besides these object-image-guided approaches, we include a focused comparison with text-guided approaches for a thorough evaluation. See Supp. C.6 for the discussion.Evaluation HOI-Swap is evaluated through both human and automatic metrics. We conduct a user study, involving 15 participants that assess 260 image edits and 100 video edits. Participants are presented with four edited results, three from baselines and one from HOI-Swap (randomly shuffled) and asked to pick the best one in terms of object identity, HOI realism, and overall quality (for image edits), motion alignment and overall quality (for video edits). Each sample is assessed by three different participants. For automatic evaluation, we employ several metrics across different dimensions: (1) HOI contact agreement and hand agreement to measure spatial alignment, following [62, 66] ; (2) hand fidelity, following [37] ; (3) subject consistency and motion smoothness from VBench [23] to evaluate general video quality. See Supp. B.3 for full descriptions.Implementation The stage-I model is trained on 512×512 resolution images for 25K steps. For stage-II training, input video resolution is set as 14×256×256, where we sample 14 frames at an fps of 7 and train the model for 50K steps. See Supp. B.4 for full details.Quantitative evaluation Table 1 compares the performance of HOI-Swap on both image and video editing tasks. For image editing, the two inpainting approaches, PBE [57] and AnyDoor [7] struggle to generate the hand correctly, as indicated by hand fidelity scores below 90%. Afford Diff [62] is developed for the HOI context and also benefits from HOI-specific training data in HOI4D. It produces high-fidelity hands but does not address the editing task adequately, resulting in low contact and hand agreement with the source image. Our proposed HOI-Swap excels in both aspects, achieving superior performance in agreement and fidelity scores. In the human evaluation, HOI-Swap is consistently favored over the other three approaches, achieving a preference rate of 72.1%. For video editing, HOI-Swap outperforms all competing methods across all metrics, including general video quality metrics (subject consistency and motion smoothness) and HOI edit metrics (contact agreement, hand agreement and hand fidelity). It significantly surpasses the three baseline approaches, winning the user study with a selection rate of 86.4%. Recognizing the inherent challenges of the video editing task, we included a survey question asking users if "All edits are of very low quality; none successfully swap the object." This response was selected at an average rate of 10.9%. See Supp. C.3 for details. adjusting hand-holding patterns based on the reference object's properties (rows 1 and 2, with source images from HOI4D [35] ). Moreover, HOI-Swap serves as a general object-swapping tool (not limited to HOIs) and can seamlessly swap in-contact objects in cluttered scenes (rows 3 and 4, with source images from EgoExo4D [15] ). The final row demonstrates HOI-Swap's zero-shot generalization capability, using source images from EPIC-Kitchens [8] . Despite the out-of-distribution samples, HOI-Swap successfully aligns the reference mug handle with the source's hand context, whereas baselines like AnyDoor [7] fail to establish this connection. See Supp. C.1 for more examples.For video editing (Figure 4 right), the source video depicts a hand rotating a toy car. The per-frame approach struggles with temporal consistency of the object identity and produces unnatural HOI motions. AnyV2V [28] fails to propagate the first-frame edit through subsequent frames, leading to inconsistencies. VideoSwap [17] preserves the overall look of the source video but fails to swap the reference object (i.e., the green toy car). In contrast, HOI-Swap delivers video edits that realistically depict the HOI and faithfully follow the source video's motion. Beyond this specific example, we note that baseline methods generally perform poorly on our in-contact object swapping task across various examples (see the project page video for more results). Despite our best efforts to reproduce their results using the official code and adjusting hyperparameters according to the provided guidelines, the performance of these models remained notably suboptimal. We observe similar issues with state-of-the-art text-guided approaches (Supp. C.6). This persistent underperformance highlights a big gap in current (otherwise quite powerful) diffusion models in their ability to accurately model the intricacies of HOIs, indicating great opportunities for further advancements in this field.Ablation study on motion points sparsity HOI-Swap can generate videos with different degrees of motion alignment with the original video. Figure 5 compares the generated results when sampling no points versus full points. Here the source video depicts a closing trash can lid action. In row 2, when no points are sampled (resulting in no warping), the generated video does not follow the closing lid motion but instead depicts an action of putting the lid into the bin; the visible hand motion outside the inpainted region provides context, guiding the model to generate a plausible sequence with the hand moving downward. Conversely, using all motion points produces videos that closely align with the source video's motion. See Supp. C.4 for follow-up discussion on this.One-stage vs. two-stage design Next we verify the efficacy of our two-stage design, where we train a one-stage model variant that takes the reference object image and the masked frame sequence as input and outputs the edited video. Figure 6 shows both qualitative and quantitative comparisons. The one-stage approach does not yield satisfactory results, specifically failing to preserve the reference object's identity. This limitation arises from the challenge of simultaneously addressing both spatial and temporal alignment, thereby reinforcing the advantage of our two-stage design.  
limitations and future work: 
 The problem of swapping in-contact objects in videos poses significant challenges. While HOI-Swap marks an important first step towards addressing these complexities, we recognize its current limitations. Specifically, we identify three areas for improvement: (1) generalization to new objects. HOI-Swap capably swaps new object instances with HOI awareness (e.g., swap an unseen mug with a differently-shaped bowl). More challenging scenarios involve presenting very different unseen reference object images for swapping. For example, the model needs to accurately depict a hand holding scissors, even though scissors were never part of the training dataset. This scenario requires the model to equip "world knowledge" that extends beyond its training data, enabling it to understand and realistically model how hands interact with a broader variety of objects. (2) generalization to long video sequences with complex HOI. The two-stage pipeline of HOI-Swap is motivated by the observation that HOI interactions remain stable throughout a short video clip. For instance, for a picking up mug sequence, replacing the mug with a bowl can be reliably done in one single frame and propagated across the remaining frames. However, for longer sequences, it is possible that an object undergoes multiple distinct hand interactions that change dynamically over time. This complexity necessitates the development of methods that can capture and model these varied HOI interactions across time. (3) controllability. Our proposed controllable motion guidance allows HOI-Swap to freely choose the degree of motion alignment with the source video in editing. One future work direction is to enhance this controllability with spatial support, allowing the model to specify which regions of the source video should be targeted for motion transfer. See Supp. D for more details on HOI-Swap's limitations and failure modes, including visual examples. 
conclusion: 
 Recognizing the limitations of current diffusion models in effectively capturing HOIs, we introduce HOI-Swap, a novel approach for swapping hand-interacting objects in videos based on one reference object image. HOI-Swap consists of two stages: the first seamlessly integrates the reference object into a single frame, while the second propagates this edit across the entire sequence, with controllable motion alignment. Both qualitative and quantitative evaluations demonstrate that HOI-Swap produces realistic video edits with accurate HOIs that align well with the source content. In all, this work broadens the capabilities of generative models in video editing and represents the initial step towards solutions that can adeptly handle the intricacies of complex and dynamic HOIs in videos. Additionally, we test the model's zero-shot editing capabilities on four EPIC-Kitchen [8] videos, which depict in-the-wild kitchen scenarios with often cluttered backgrounds, and four videos from TCN Pouring [48] , with two being egocentric and two exocentric, featuring pouring motions. This evaluation encompasses a variety of challenging scenarios to thoroughly assess model performance.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Please see main paper Section 4.1 and Supp. Section B for full experimental setup of HOI-Swap.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Please see Table 4 for error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discussed the potential negative societal impact of this work when being used to create misleading content in Supp. Section E. We also discussed possible solutions to mitigate the misuse.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The focus of our paper is on video editing, specifically swapping in-contact objects in video sequences. This application does not involve the generation of misleading information, nor does it use datasets or models prone to misuse (we train our models using HOI4D and EgoExo4D data).Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: Please see main paper Section 4.1 and Supp. Section B.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We implement the baselines described in Section 4.1 using their official code. Baselines including PBE, AnyDoor and AnyV2V are adopted in a zero-shot manner. Afford Diff utilizes the same training data source (HOI4D) as ours. We do not provide additional layout or hand orientation information to Afford Diff. VideoSwap [17] , a one-shot-tuned video editing approach, is trained on the source video using the default setup in their official repository. AnyDoor is recognized as the best image editing method and is integrated into the two video editing baselines, per-frame and AnyV2V.Note that these baseline methods impose more demanding test-time requirements compared to ours. PBE and AnyDoor both necessitate precise segmentation masks of the source object, while HOI-Swap only requires a bounding box during inference. In addition, VideoSwap requires users to click semantic points for every video, provide a text prompt describing the video, and train a specialized object model using a few reference images. This extensive prepossessing limits its scalability and is a contributing factor to the smaller size of our video evaluation set. In contrast, HOI-Swap is fast on inference, with RAFT optical flow extraction completing in just a few seconds per video, making it efficient and practical for real-world applications.Automatic evaluation For automatic evaluation, we employ several metrics across different dimensions: (1) To measure how well the generated content aligns spatially with the source image, we use HOI contact agreement and hand agreement. Following [62, 66] , we utilize the in-contact state label provided by an off-the-shelf hand detector [49] . The HOI contact agreement score is determined by checking if the in-contact label of the generated frame matches the source frame. Additionally, we calculate mIOU between the generated and source hand regions using the detector's bounding boxes for hand agreement. (2) To evaluate how realistic the generated hand is, we follow [37] and assess hand fidelity based on the hand detector's confidence score. (3) To evaluate video quality, we consider subject consistency (DINO feature similarity across frames) and motion smoothness (determined by motion priors from a video frame interpolation model [33] ) as proposed in VBench [23] . Note that we exclude object agreement from the automatic evaluation due to its inaccuracies. Often, the generated hand occludes some object regions, leading to lower similarity scores compared to simply placing the reference object in an area with no or partial hand presence. This discrepancy does not accurately reflect HOI realism and generation quality. To ensure a comprehensive evaluation, we incorporate questions in our user study to address this dimension.We conducted a user study with 15 participants who assessed 260 image edits and 100 video edits randomly selected from the evaluation set. The human evaluation survey interfaces for image and video editing are shown in Figure 7 and Figure 8 , respectively. Each survey began with general guidelines on the evaluation process, followed by examples of high-quality image or video editing results and good motion alignment for videos. For each question, participants were presented with four edited results: three from baselines and one from HOI-Swap, randomly shuffled. Each image or video sample was evaluated by three different participants to minimize bias.For each image editing question, participants were asked to select the best one or two edits based on the following three evaluation dimensions:1. Object Identity: The ability to preserve the identity of the reference object.For data preprocessing, we apply several transformations to the reference object image: random rotations with angles ranging from -90 to 90 degrees, random perspective changes, and random horizontal flips. We crop and center the source frame using a variable ratio between 0.3 and 0.6, determined during training to introduce variability; this ratio is defined as the size of the bounding box relative to the image size. For stage-I training: image resolution is set as 512×512. We train the model for a total of 25K steps with a learning rate of 1e-4 and a batch size of 8. We finetune the entire 2D UNet for image editing. For stage-II training: input video resolution is set as 14×256×256, where we sample 14 frames at an fps of 7. The model is trained for a total of 50K steps with a learning rate of 1e-5 and a batch size of 1. We finetune the temporal layers of the 3D UNet. A classifier-free guidance dropout rate of 0.2 is employed for all stages. Training for each stage takes about 3 days on one 8-NVIDIA-V100-32G GPU node. We set sampling points sparsity value as 50% for all quantitative evaluations.Beyond the examples shown in Figure 4 of the main paper, we provide additional qualitative image editing results of HOI-Swap in Figures 9 and 10 . HOI-Swap excels in both structured scenarios with relatively clean backgrounds (Figure 9 , source frames from HOI4D and TCN Pouring), and challenging in-the-wild scenarios featuring cluttered backgrounds (Figure 10 , source frames from EgoExo4D and EPIC Kitchens). Compared to the baselines, HOI-Swap seamlessly integrates the reference object into the target scene, demonstrating exceptional HOI awareness and the ability to accurately reorient the reference object to align with the scene context. Moreover, it also performs well in general object-swapping settings where no hand is involved, as shown in rows 3, 5, and 6 of Figure 10 . Please refer to our project page for more qualitative results of video editing.Table 1 of the main paper reports video editing results when videos are split based on object instances. In addition, we conduct experiments with two splitting ways: (1) across different subjects, and (2) across different actions. The results are reported in Table 2 . HOI-Swap consistently outperforms baseline approaches across different data splitting settings, demonstrating its effectiveness. Furthermore, we provide a breakdown of video editing results in Table 3 , separating results from in-domain videos (HOI4D) and out-of-domain videos (EPIC-Kitchens & TCN Pouring). We observe that the performance gain of HOI-Swap is higher for in-domain videos than out-of-domain videos. Future work could explore extending HOI-Swap's generalization capabilities. Table 4 provides the breakdown of the evaluation dimensions for our HOI-Swap vs. baseline models on both image and video editing tasks. In terms of all metrics for both image and video editing, our HOI-Swap's editing results are consistently preferred by humans. Note that an exception is in object identity, where our HOI-Swap does not surpass Afford Diff. This is because Afford Diff focuses on synthesizing hands on fixed objects, directly copying and pasting the object image from the given reference object image.Table 4 : Breakdown of user study evaluation metrics. For image editing, we consider metrics including object identity, HOI realism, and overall quality. For video editing, we consider metrics including motion alignment and overall quality. Each value represents the frequency (expressed as a percentage) with which participants prefer that model considering a particular evaluation dimension. Standard deviations are calculated across participants to illustrate preference variability. For video editing, users were given an additional option to indicate if they found all edits unsatisfactory, which has an average rate of 10.9%. Note that our HOI-Swap does not surpass Afford Diff in the ability to preserve object identity. This is because Afford Diff focuses on synthesizing hands on fixed objects so the model will directly copy and paste the object image from the given reference object image. C.4 Ablation study DINO vs. CLIP encoder Stage I and stage II of our pipeline employ the DINO and CLIP encoders, respectively, to align with their specific objectives. Stage I, focused on swapping the reference object within a single frame, benefits from the DINO encoder due to its enhanced ability to capture "objectness" compared with the CLIP encoder. The main emphasis of stage II is to transfer motion from the source video, and a CLIP encoder is adopted to provide scene context for generating the video. We additionally conduct experiments to explore the possibility of using a DINO encoder for stage II. As reported in Table 5 , the two encoder variants perform similarly.Ground truth vs. estimated object masks While it is generally assumed that users will provide the ground truth bounding box or segmentation mask of the object they wish to replace during Edited Frame Region to be inpainted inference [57, 7] , we acknowledge the potential of incorporating automatic segmentation methods to reduce user effort. Following this, we applied SAM-2 [44] to identify bounding boxes on test videos as an alternative to manually providing ground truth. This method requires just a single click inside the object in the initial frame, followed by SAM-2 automatically tracking the target object. The quantitative comparisons are presented in Table 6 . While there is some degradation across three HOI metrics, we believe this feature is valuable for downstream applications as it greatly eases the input requirements and improves user convenience. Note that the baseline approaches have more stringent input requirements than HOI-Swap, e.g. need precise object segmentation masks or additional text prompts. Even with the use of automatically generated masks, HOI-Swap demonstrates its great advantages over the baselines. Masking strategy In stage I, we mask the source frame with a square bounding box, as opposed to using the source object's segmentation mask as in [57] . This masking strategy not only directs the model to fill the predefined masked area but also to generate plausible hand-object interactions aligning with the source frame's hand position and the reference object. To demonstrate this point, we experimented with a variant of the stage I model that uses the original object's segmentation mask instead. As illustrated in Figure 12 , this variant struggles with grasp changes when the reference and source objects differ, thereby reinforcing the effectiveness of our chosen approach.Editing frame selection One question naturally arising from our two-stage editing pipeline is determining which frame from the video sequence to edit. Recall that our training strategy (Section 3.3) prepares the stage-II model to be any-frame-conditioned, bringing maximum inference flexibility. Here, our key insight is to choose a hand-object contact frame for editing due to two main advantages: (1) Hand-object contact frames are particularly challenging as they require generating correct interaction patterns (e.g. grasps) compared with non-contact frames; prioritizing them ensures that the model can handle the most complex scenarios. (2) Hand-object contact frames offer the most comprehensive scene context for the stage-I model, aiding in spatial alignment.To illustrate, in Figure 11 , we compare video editing results when choosing the 1st, 4th and 7th frames in a sequence of picking up a mug. First, it is evident that the 7th frame is the most challenging to edit since it requires precise handling of the grasp region, while the 1st and 4th frame involve no or little hand interaction. Next, while single-frame edits for the 1st and 4th frame look reasonable (successfully swapping the mug), the lack of hand context in these frames leads to incorrect orientation of the mug handle. Consequently, this causes failures or imperfections in generating the subsequent frames 10 and 13, where the hand is unrealistically holding the mug.Motion points sparsity Following the discussion in Section 4.2 of the main paper, we present a comprehensive analysis of the impact of sampled point sparsity. Figure 13 illustrates two swapping cases, involving the motion of closing a trash can lid. For the left video, the two trash cans differ greatly in shape and closing mechanism. Using all points confuses the model, resulting in an incorrect lid-closing motion. Conversely, using no points generates a different motion of putting down the trash can rather than closing the lid. In this example, using 50% of the points yields the most plausible edit.For the right video, as discussed in the main paper, due to the similarities between the two objects, replicating the full motion by adopting 100% points is effective. In all, HOI-Swap allows users to flexibly decide the extent of motion transfer when generating content, accommodating different scenarios and object characteristics. As described in Section 3.3, we perform uniform sampling of points within the bounding box region.When the new object differs greatly in shape from the original, sampled points might incorrectly capture motion-background points could be assigned foreground motion, and vice versa. A direct solution to mitigate this mismatch is to reduce the sampling density. Nevertheless, we find that HOI-Swap demonstrates considerable robustness even when a large number of points are sampled, adeptly managing potential discrepancies within the warped sequence V warp . As shown in Figure 14 , with 50% points sampled, the visualized V warp reflects the original object's shape to a certain extent, yet the model effectively distinguishes between background and the foreground object, recognizing the object as a whole and thereby generating plausible videos.We identify two potential reasons for the model's effectiveness. First, our training strategy helps enhance the model's capability to discern between foreground and background elements, since we intentionally sample points within the bounding box region rather than solely within the object's segmentation mask. Consequently, the model is trained with a mix of in-object points and background points, preparing it to handle background points that may be sampled outside the new object region during inference. Additionally, the relative constancy of background and clear distinction between foreground and background contributes to the model's robustness, allowing it to recognize the object as a whole and accurately move it even when some regions inaccurately carry motion. and Rerender-a-video (Rerender) [59] . To evaluate these latest models on the object swapping task, we describe the reference object in text and prompt the models to replace the original object in the video. These approaches are unable to alter the shape of the bowl and fail to swap the original bowl with a kettle as required.To understand how the latest text-guided diffusion models perform on this task, we provide qualitative comparison of HOI-Swap with three models: Pika [30], Runway [10] , and Rerender A Video [59] . We adopt the text prompt: "change the object in the video to a kettle while keeping background and motion the same", which proved most effective compared to more complex prompts. As shown in Figure 15 , these approaches edit the video in a way that preserves the original object shape, failing to generate the new object or adapt the HOI patterns. These results underscore the unique capabilities of HOI-Swap.One area for improvement stems from the sampling region discussion in Section C.5, with sampled points potentially carrying wrong motion when object shape changes, particularly when a large number of points are sampled. To further refine our model's performance, a future direction is to explore fine-grained spatial control in motion alignment. Specifically, concentrated sampling could provide the model with the precise ability to modulate specific regions of an object. These enhancements are anticipated to augment the model's adaptability and accuracy in editing complex HOI sequences.Another direction is the development of an automated method to determine the sparsity of motion points based on the object swapping scenario and the motion complexity in the original video. This would allow the model to autonomously decide the extent of motion transfer necessary when generating new content.In addition, we present failure cases in Figure 16 . The left example shows a stage-I failure, where HOI-Swap fails to achieve accurate spatial alignment. The right example illustrates a stage-II failure, where noise in the warped sequence impedes model's understanding of the reference object, leading to wrong motion patterns.Finally, we emphasize that the problem of swapping in-contact objects in videos poses significant challenges, particularly with complex HOI sequences. We recognize the limitations of HOI-Swap in its current stage and view this work as an initial step towards resolving these challenges. Looking ahead, we aim to incorporate more precise spatial and motion controls, and expand HOI-Swap's capabilities to handle longer video sequences with more intricate HOIs.This project aims to equip the community with a powerful tool for swapping customized objects into videos, particularly improving the realism of hand-object interactions. However, there is a potential risk that malicious users could misuse this technology to produce deceptive videos involving real-world individuals, potentially misleading viewers. This issue is not exclusive to our approach but is a common concern across various video editing and generating modeling techniques. To mitigate such risks, one possible solution is to introduce subtle noise perturbations to published images to disrupt the customization process. Furthermore, implementing invisible watermarking in the generated videos could help prevent misuse and ensure proper attribution of the content.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We clearly describe our main claims such as the findings, contributions, and results in the abstract and introduction. They match our idea, methodology, and experimental results presented in the paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Project webpage: https://vision.cs.utexas.edu/projects/HOI-Swap. 


Decision: The paper studies the problem of swapping objects in hand-object interaction videos. The authors present a diffusion-based video editing framework. Comprehensive evaluations show that HOI-Swap outperforms existing methods, delivering high-quality video edits. Three reviewers think that their concerns are mostly addressed, and one reviewer further proposed that this paper should clarify its downstream scenarios further explain its capability boundaries, and raise the scores. After reading this paper, reviews, and responses, the AC agrees that this work made a good contribution.

Reviews: review: paper_summary: paper_summary: The work considers the task of swapping the objects in ego-centric short clips with hand-object interactions. The manuscript claims to introduce this sub-task within the field of generative video editing. Concretely, HOI-Swap starts from RGB video, object area bounding box, and an image of the target object, and generates a new video, transferring hand movement to the new object, possibly adapting to the changing functionality of the object. The method has two stages: inpainting a single frame, and extending it to the whole sequence ensuring consistency. HOI-Swap is thoroughly evaluated against recent video and image editing methods, also the work demonstrates qualitative examples of generalization to novel datasets.
strength_weakness: strength_weakness: A minor weak point is the lack of detailed discussion on downstream applications of the model. The Introduction section touches on this topic, however, an example of possible demos or a deeper discussion of possible usages of the model could help to better motivate the importance of the introduced task and the proposed method. 

Another possible area for improvement is automatic input mask annotation. Currently, the method requires a bounding box of the object as an input, however, with current advancements in the segmentation methods this requirement can be alleviated (or at least quantitatively compared to the GT bbox usage).
questions: questions: 1) Stages 1 and 2 employ different image encoders (DINO and CLIP), what is the motivation for this choice?
2) Lines 212-220 discuss the ability to vary the number of control points sampled for motion guidance. What is the default value used for quantitative evaluations?
limitations: limitations: A detailed discussion of limitations is present in the Appendix with complementary examples in a supplementary video.
paper_summary: paper_summary: This article focuses on proposing a two-stage network HOI-Swap, a video editing framework designed for precise object edits with HOI awareness.To address the problem of real perception of HOIs as well as spatial and temporal alignment with the original video, HOI- Swap first stage focuses on solving HOI awareness and establishing spatial alignment by training an image restoration diffusion model to swap objects in one frame. The second phase deforms video sequences from edited frames by tracking inter-frame points with randomly sampled optical flow, and trains a video diffusion model to generate new videos.
strength_weakness: strength_weakness: 1. Limited innovation. The methodology of this paper is a combination of existing work and is more like a technical report than an academic paper.
2. Lack of dynamic HOI awareness transfer. The second stage does not embody HOI interaction awareness. The single-frame propagation only propagates the motion information of the object and does not embody how the HOI-aware information is conveyed in the video sequence.
3. Insufficient experiments. This paper only demonstrates HOI awareness in single-frame image restoration, and in the video aspect, either the scene with little change in hand poses or the scene with little change in object characteristics is demonstrated, but for the complex video sequences with large changes in object characteristics and the need for dynamic changes in hand pose, the paper does not give the relevant effects to demonstrate.
questions: questions: 1.line 174, this paper suggests that data augmentation techniques such as flipping, rotating, perspective transformations, etc. are performed due to the difference in size and pose of the target and reference objects, so why not use scaling techniques to bridge the gap between large and small objects?
2.Line205, this paper proposes to apply the largest bounding box as mask to the whole video sequence, so for the case of a relatively large range of object movement, will a larger mask range lead to the proportion of sampling points that does not truly reflect the proportion of extracted object features and motion information？(For example, if the target object only occupies 1/4 size of the mask in a certain frame, then a uniform sampling ratio of 50% is also difficult to extract the object motion information).
3. Why not continue the use of DINO for the first stage network instead of switching to CLIP in the second stage network?
limitations: limitations: Yes
paper_summary: paper_summary: This work presents a novel approach for object insertion in hand-object interaction scenarios. The proposed approach consists of two stages: image-based editing to for precise alignment of hand with the inserted object, and video-based editing for motion alignment with the original video. The model is trained in a self-supervised manner and also imparts a varying level of controllability to adjust the degree of motion alignment based on object changes. Extensive experiments on HOI4D, EgoExo4D, EPIC-Kitchens and TCN Pouring datasets show the effectiveness of the proposed approach in various scenarios.
strength_weakness: strength_weakness: - Several details about the experimental setup are missing from the main paper. These details are important to understand the scope of the claims in the paper. While the supplementary contains more details, it'd be helpful to have more clarity on the following aspects:
    - For evaluation on HOI4D & EgoExo4D, how is the test split created? Do the held-out videos contain different object categories or different instances from the same category or different subjects performing the experiments or different actions being performed? Having quantitative experiments in these different settings would help in understanding the benefits and limitations of the proposed approach.
    - Do the results in Tab.1 span all 4 datasets? Again, it'd be helpful to have a breakdown of the results in different settings to understand where the proposed approach is most effective.
- Are all the baselines trained in the same setup as the proposed approach? I understand that it might not be feasible to implement/train all baselines due to differences in architecture and compute requirements, but it'd be useful to clearly describe the protocols followed for each baseline. Are these baselines retrained or used in a zero-shot manner? For example, AffordDiff [62] only takes a single RGB image of the object as input and generates the HOI image, it does not take the reference video into account so it is expected that the object orientation may not be consistent with the video (L43-50). AffordDiff also allows for controllability in the layout & orientation of the generated hand so it can be modified to match the hand in the reference video. I'm not asking to reimplement AffordDiff to adapt to this new setting, but it'd be useful to have these details in the main paper to understand the differences between baselines and the proposed approach.
- While the 2-stage decomposition of the approach makes sense, it'd also be useful to quantitively verify that it is indeed better than the 1-stage analog. For example, in Fig. 3, Stage-2 can take the DINO encoding of the reference object instead of the generated image from Stage-1 to create a 1-stage approach. This is not required for rebuttal, but a suggestion to validate the 2-stage approach.
questions: questions: I need clarifications on the following aspects to better understand the results (more details in the weaknesses above):
- For evaluation on HOI4D & EgoExo4D, how is the test split created? Do the held-out videos contain different object categories or different instances from the same category or different subjects performing the experiments or different actions being performed?
- Do the results in Tab.1 span all 4 datasets? It'd be helpful to have a breakdown of the results in different settings to understand where the proposed approach is most effective.
- Are all the baselines trained in the same setup as the proposed approach? It'd be useful to clearly describe the protocols followed for each baseline. Are these baselines retrained or used in a zero-shot manner?
limitations: limitations: L330-332 in the main paper defers the limitations and failure modes to the supplementary without providing any insights. Please add these details to the main paper.

---
I have read all the reviews and the rebuttal. I thank the authors for providing additional clarifications, which help me understand the paper better and I have increased my score to Weak Accept.
paper_summary: paper_summary: This paper presents HOI-Swap, a diffusion-based video editing framework for object swap editing. HOI-Swap involves two stages. In the first stage, the authors train an image-editing model to swap object in one frame. In stage II, the authors first warp a video from the edited frame using the optical flow sampled from the source video. Then a video diffusion model is trained to reconstruct the whole video from the warped frame sequences. Through this two-stage approach, HOI-Swap successfully addresses three main challenges in HOI video editing: (a) HOI-aware capabilities, (b) spatial alignment of the object with the hand, and (c) temporal alignment with the source video. Experiments demonstrate that HOI-Swap outperforms both image-editing and video-editing baselines.
strength_weakness: strength_weakness: 1. Stage I does not seem to address HOI-awareness explicitly. The authors should explain why the image-editing model can perceive HOI.
2. The authors did not mention the generalizability of their method across unseen kinds of objects.
questions: questions: 1. How can the image-editing model in stage I perceive HOI? The authors should provide more explanation and maybe some more ablation studies to support their results.
2. The whole pipeline seems applicable to general object-swap tasks. Have the authors tested the model on other objects?
3. Are the backgrounds in the test set all seen by the model during training?
limitations: limitations: The authors have adequately addressed the limitations.

Comments: title: Thanks for the clarifications
Response to Reviewer rdgi's further comment
Reply by Reviewer ih2o
Reminder: author-reviewer discussion will end soon!
Author-Reviewer Discussion Begins!
comment: Thanks to the authors for providing a rebuttal. The authors' replies on scaling as a data augment technique and mask region selection as well as additional experiments evaluating CLIP vs DINO as an encoder for stage 2 answered my confusion. 

1) Novelty.  I agree that this paper primarily focuses on a new task: video HOI editing. However, when it comes to the more significant aspect of HOI awareness, I still believe that there isn't much novel design.
2) Dynamic HOI Awareness. As written in the rebuttal, HOI-Swap lacks the ability to handle long video sequences with more complex HOI variations, which is an important direction for video editing.
3) Object Shape Variations. I am more interested in examples where the hand-object interaction pose varies greatly due to different categories. I agree with the validity of replacing the kettle with a bottle and a bowl-like object in Fig. 1 of the main paper, but the poses of the grasped objects are similar.

I agree with Reviewer ih2o that this paper should clarify its downstream scenarios and further explain its capability boundaries.
Considering that the authors solved some of my confusion and the paper does propose a new task, I raised the score to Borderline Accept.
Thank you for your insightful feedback and valuable support! We are pleased to confirm that we have fully addressed your concerns. We will make the necessary updates to the manuscript in light of your thoughtful suggestions.
Thank you for your insightful feedback and valuable support! We are pleased to confirm that we have fully addressed your concerns. We will make the necessary updates to the manuscript in light of your thoughtful suggestions.
I appreciate all the clarifications provided by the authors. I have increased the score to Weak Accept and suggest to include these additional details in the final version.
Thank you for your insightful comments and for clarifying your question regarding general object-swap tasks. In the experiments, we indeed include a diverse set of images to evaluate our model comprehensively. As noted in Ln 259-261, our evaluation involves both HOI-contact and non-HOI-contact scenarios:

+ Our editing benchmark comprises 5,000 test images, of which 20.1% represent non-HOI-contact scenarios, highlighting our model's ability to handle general object-swap tasks where hand-object interaction is absent.
+ For qualitative results, row 3 and 4 of Figure 4 in the main paper, row 3, 5 and 6 of Figure 9 in Supp. showcase general swapping scenarios without human hands, emphasizing our model's versatility across different contexts. 

We appreciate your suggestion and will expand on this discussion in the updated manuscript. Thank you once again for your thoughtful review.
Thank the authors for their response. I think most of my concerns are addressed.

I want to clarify Question 2. By "general object-swap tasks" I mean object-swap tasks without HOI and "other objects" means different categories of objects. The authors have addressed my concerns on object category-level generalization. I hope the authors include more discussion of the general object-swap task in final manuscript.

Generally speaking, this is an interesting and novel paper. For now, I will keep my initial score.
I thank the authors for providing the rebuttal. The additional experiments (evaluation of CLIP vs DINO as an encoder for stage 2 and evaluation of using non-GT mask predicted on the fly) clarify my concerns and strengthen the work.

After considering the authors' replies and other reviews, I have decided to increase my score to accept. The work tackles a novel problem, providing a solid baseline for future research.
Dear reviewers, 

The authors have responded to your questions and comments. Please propose your post-rebuttal comments. 

Best,
Your AC
Dear authors,

Thanks for your hard work and detailed responses.

Dear reviewers,

the authors have responded to your questions and comments, please read them and further provide feedback: Whether your concerns addressed? After reading all the reviews and responses, are there more questions about the clarity, contributions, results, etc?

Thanks!

Best, Your AC