HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness
Authors: Zihui Xue
Mi Luo
Changan Chen
Kristen Grauman
Venue: NeurIPS 2024 poster (2024)
Keywords: video editing
hand-object interaction
Primary area: machine_vision
TL;DR: We present HOI-Swap, a diffusion-based video editing framework, that seamlessly swaps the in-contact object in videos given a reference object image.

Abstract: We study the problem of precisely swapping objects in videos, with a focus on those interacted with by hands, given one user-provided reference object image. Despite the great advancements that diffusion models have made in video editing recently, these models often fall short in handling the intricacies of hand-object interactions (HOI), failing to produce realistic edits---especially when object swapping results in object shape or functionality changes. To bridge this gap, we present HOI-Swap, a novel diffusion-based video editing framework trained in a self-supervised manner.  Designed in two stages, the first stage focuses on object swapping in a single frame with HOI awareness; the model learns to adjust the interaction patterns, such as the hand grasp, based on changes in the object's properties. The second stage extends the single-frame edit across the entire sequence; we achieve controllable motion alignment with the original video by: (1) warping a new sequence from the stage-I edited frame based on sampled motion points and (2) conditioning video generation on the warped sequence. Comprehensive qualitative and quantitative evaluations demonstrate that HOI-Swap significantly outperforms existing methods, delivering high-quality video edits with realistic HOIs.

HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness
Zihui Xue
Mi Luo'
Changan Chen
Kristen Grauman FFIIR, Meta
'The University of Texas at Austin
Abstract
We study the problem of precisely swapping objects in videos. with focus on those interacted with by hands, given one user-provided reference object image. Despite the great advancements that diffusion models have made video editing recently, these models often fall short in handling the intricacies of hand-object interactions (HOI), failing to produce realistic edits especially when object swapping results object shape or functionality changes. To bridge this gap, present HOI-Swap, novel diffusion-based video editing framework trained in a self-supervised manner. Designed in two stages, the first stage focuses on object swapping in single frame with HOI awareness; the model learns to adjust the interaction patterns, such as the hand grasp, based changes in the object's 's properties. The second stage extends the single-frame edit across the entire sequence; we achieve controllable motion alignment with the original video by: (1) warping edited frame based on sampled motion points and (2) conditioning video generation the warped sequence. Comprehensive qualitative and quantitative evaluations demonstrate that HOI-Swap significantly outperforms existing methods, delivering high-quality video edits with realistic HOIs.
sequence from the stage-I
Introduction
Consider a video depicting the process of human hand picking up kettle and moving around (Figure 1). What if we want to replace the hand-interacting object in this scene with another item specified by the user-perhaps a differently-shapee kettle, a bottle, or bowl? This capability- swap the in-confact object in a video with another, while aligning with the original video's oes content-ia crucial for enhancing various real-world applications. Such functionality can transform entertainment, allowing users to create novel video content without the need to re-record or engage in labor-intensive manual editing. For example, in advertising, there may be situations where a pre-recorded video needs to adapt new sponsorship requirements by replacing soda can in the video with water bottle. Additionally, it holds significant promise in robotics, where recent results suggest generative models can reduce the reliance manually collected task-specific visual data and thereby enable large-scale robot learning [25, 58]. For example, imagine scenario where, from just a single video of mug being picked up, a generative model is able produce numerous variants of this video with diverse objects such as bottles, bowls and kettles. This capability could greatly streamline the data collection process, reducing the need for extensive manual data collection.
However, hands are notoriously challenging to work with in image/video editing [62, 66]. They pose significant hurdles in manual photoshopping and often produce unsatisfactory outputs when automated by generative models. The precise swapping of hand-interacting objects presents a unique challenge that existing diffusion models [19, 45, 1, 56], despite their advances in video editing, fail to address adequately. This difficulty arises from three main factors: the need for (a) HOl-aware capabilities, (b) spatial alignment with the source context, and (c) controllable temporal alignment with the source video's motion patterns.
Project webpage: https://visiee www.ttaaaa...m wwweeddueeeee htta//proeects/HO0 Swap.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Source (Originll Video)
HOI- generated video (A)
swap the in- contact with:
HOI- -Swap generated video (B)
HOI-Swap generated video (C)
(A)
(8)
(C)
Figure 1: We present HOI-Swap that seamlessly swaps the in-contact object in videos using a reference object image, producing precise video edits with natural hand-object interactions (HŌI), Notice how the generated hand needs to adjust to the shapes of the swapped-in objects (A,B,C) and how the reference object may require automatic re-posing to fit the video context (A).
First, consider the challenge of HOI awareness. Objects chosen for swapping often vary in their properties from the original, resulting in changes to interaction patterns. For example, illustrated in Figure (a), replacing the kettle from the original video with a bowl necessitates adjustments in the hand's grasp pattems. While many generative inpainting methods [57, 50, 36, 7, 51, 16] have been developed to insert reference objects into specific scene regions, they are generally limited to contexts where the objects are isolated-not in contact with human hand other objects-and thus lack HOI awareness. In Figure 2 (a), the two image inpainting approaches Paint by Example (PBE) [57] and AnyDoor [7] either merely replicate the hand pose from the original image, or produce unnaturally occluded hands, resulting in suboptimal and unrealistic HOI patterns.
Second, consider the challenge of spatial alignment with the original video. The reference object might appear in any arbitrary pose; for instance, in Figure 2 (b), the kettle handle on the left in the reference image, but for realistic interaction, the generated content needs to reposition the kettle handle to the right, where the hand is poised to grasp it. However, current approaches do not offer this level of control, as evidenced by the results from hand insertion approach Affordance Diffusion (Afford Diff) [62] in Figure 2 (b). Despite heing adept at generating a hand interacting with the given kettle, it does not ensure correct object placement to align with the hand and scene context in the original image, lacking spatial alignment capability.
Third, consider the challenge of temporal alignment with the original video. We highlight a crucial observation in this problem: the motion information an HOI video sequence is closely tied to the object's is characteristics (such as its shape and function). This means that when swapping objects, not all motions from the source video are appropriate or transferable to the new object. For example, Figure 2 (c) shows a hand closing a trash can, alongside an HOI-Swap edited image where the original can is replaced with one that is differently shaped and functions differently. Ideally, the generated content should reflect the motion of closing the lid, yet it may not replicate the exact motion from the source video due to these differences. Conversely, Figure (first row) depicts a scenario of swapping one kettle with another. Here, the objects undergo only slight shape changes, allowing the generated video to closely follow the source video's motion. While there are varying degrees to which object changes can affect the original motion, current video editing approaches [42, 12, 10, 59, 21, 6, 5, 34] adhere to a rigid degree of motion alignment, often targeting 100% adopting conditional signals like optical flow or depth sequences that encode substantial object information. Consequently, they lack the controllability to adjust the degree of motion alignment based on object changes.
To address these challenges, introduce HOI-Swap, video editing framework designed for precise object edits with HOI awareness. We approach the challenge by posing it as a video inpainting task, and propose a fully self-supervised training framework. Our innovative approach structures the editing process into two stages. The first stage addresses HOI awareness and establishes spatial alignment, by training an image inpainting diffusion model for object swapping in one frame. The second stage propagates this one-frame edit across the remaining frames to achieve temporal alignment with the source, For this, we propose to warp a video sequence from the edited frame using randomly sampled points tracked across frames with optical flow, and train a video diffusion model that learns to fill in
2

---

awareneds
all ginmentt with source
AnyDoor
Afford Dif
HOK-Swap
Sharts
Sculice
spatial alignment with
swapping Iame
AnyDoos
Aflond Cift
HO HOH-Swap
Editdd Froma
We highlight three challenges for the in-contact object swapping problem: (a) HOI awareness, where the model needs to adapt to interactions, such changing the grasp to realistically accommodate the different shapes of the kettle vs. the bowl; (b) spatial alignment with source, requiring the model automatically reorient objects, such as aligning the blue kettle from the reference image to match the hand position in the source; (c) temporal alignment with source, necessitating controllable motion guidance capability, essential when swapping objects like trash can with differently shaped and functioning reference, where not all original motions are transferable desirable. In (a) and (b), compare HOI-Swap's edited images with Paint by Example (PBE) [57], AnyDoor [7]. and Affordance Diffusion (Afford Diff) [62].
Figure 2:
the sparse and incomplete warped sequence. Our approach thus enables controllable motion alignment by varying the sparsity of sampled points. During inference, users can modulate this number based fewer no points for significant shape or functional alterations,
on the object's changes- and more points for minor changes to closely replicate the source video's motion. HOI-Swap is evaluated on both image and video editing tasks, consistently producing high-quality edits with realistic HOIs. greatly surpasses existing editing approaches in both qualitative and quantitative evaluations, including user study. By extending the capabilities of generative models into the largely unexplored domain of HOI, we hope that HOI-Swap opens new avenues for research and practical applications in this innovative field.
2 Related Work
Generative Models for Image Editing Recent advances in diffusion models [19, 45] have signifi- cantly enhanced the capabilities of image editing. Predominant models [38, 18, 2, 26, 53, 24] use text as guidance, which, despite its utility, often lacks the precision needed for exact control. In response. a growing body of studies have begun to explore the use of reference images as editing guidance. Customized approaches like Textual Inversion [11] and DreamBooth [47] are designed generate new images of specific object given several of its images and relevant text prompt. However, these methods require extensive finetuning for each case and lack the ability to integrate the object into another user-specified scene image. More closely related to our task, few approaches aim to seamlessly blend a reference object [57, 50, 36, 7, 51, 16] or person [29] into a specific region of a target scene image. However, as demonstrated in Figure 2 and detailed in Section 4, these methods prove inadequate, often producing images with unnatural HOIs, such as missing fingers, distorted hands, or oddly shaped objects. These issues reveal shortcomings in current generative models, and motivate the development of HOI-Swap.
Generative Models for Video Editing With the advent in diffusion-based text-to-image and text- to-video generation [45, 1, 13), many efforts explore extending pretrained diffusion models for video editing, employing zero-shot [12, 59, 42, 4, 60, 28], or one-shot-tuned learning paradigms [55, 32, 17, 67]. However, these methods require extensive video decomposition costly per-video fine-tuning, with processing times ranging from several minutes to multiple hours on high-end GPUs for a single edit, which curtails their usability in practical creative tools. Another line of work [10,6 6, 34, 54, 63, 41] adopts a training-based approach, where models are trained on large-scale datasets to enable their
immediately effective editing tools during inference. Our work falls into this training-barn based paradigm.
Similar to image editing, the majority of video editing approaches [1, 13, 59, 42, 4, 60, 55, 32) rely on text as editing guidance. Since text prompts may not accurately capture the user's intent [67, (4,41], for our task, using object image provides precise guidance. In terms of motion guidance,

---

most video editing approaches [42, 12, 10, 59, 21, 6, 5, 34, 63] utilize per- per-frame structural signals extracted from the source video, including depth map, optical flow, sketches, or canny edge sequences. facilitated by well-trained spatial-control image diffusion models such as ControlNet [65], T2I- Adapter [39] and Composer [22]. Since these structural signals inherently encode the shape of the original object, they are unsuitable for edits involving shape changes [41, 17]. Recent works explore shape-aware video editing, by the use of Layered Neural Atlas [32], modifying the source video's 's optical flow sequences [41], or establishing correspondence with sparse semantic points [17]. However, all these approaches overlook the impact of object changes on motion and only enforce a fixed degree of motion alignment. In contrast, HOI-Swap introduces flexibility by allowing users to adjust the sparsity of sampled points during inference, providing precise motion control.
Generating Hand- Objeet Interactions There is growing interest in generating plausible HOIs [20, 31,62, 61, 9, 66, 64]. Affordance Diffusion [62] inserts a synthesized hand given single object image. Similarly, HOIDiffusion [66] aims to create HOI images conditioned on a 3D object model along with detailed text descriptions. Meanwhile, other approaches in the 3D domain focus generating realistic 3D HOI interactions from textual descriptions [9], synthesizing grasping motions [64] or reconstructing 3D HOIs from real videos [61]. Despite these advancements, the task of accurately swapping in-contact objects in videos remains unexplored. Our work directly addresses this gap.
3 Method
We first formulate the problem and provide an overview of the two-stage HOI-Swap Section 3.1. Section 3.2 and Section 3.3 detail the first and second stage of HOI-Swap, respectively.
3.1 Task Formulation and Framework Overview
Given a source video V (I,) consisting of N frames (where I, represents the i-th frame), binary mask (bounding box) sequence {M:j that identifies the region of the source object to be replaced in V, and an image of the reference object Iref, the objective is to generate a modified video V", that seamlessly swaps the original object in V with ITef_ I
We create a fully self-supervised training approach, necessitated by the impracticality of collecting paired videos (),, )*) We pose the problem
video inpainting task. Specifically, from the original training video V and mask M, we derive a masked video yr", accompanied by set of reference object images {Zref]. During training, the model takes the masked video sequence y and an image In of the same object originating from different random timepoint (and hence varying pose and/or viewpoint) in the same training video to reconstruct V. During inference, the model is given a bounding box-masked sequence combined with various object images test its swapping capability.
As outlined in Section I, the task presents three main challenges: HOI awareness, spatial and temporal alignment. In response, we propose a two-stage framework that decomposes the complexity. The first stage integrates the reference object into a single frame, targeting HOI awareness and establishing spatial alignment initially. The second stage propagates the one-frame edit across the entire video sequence, focusing on temporal alignment and carrying forward the spatial alignment from the first stage. This structured approach effectively lightens the model's s generation load. In terms of which frame to select from the input video for stage-I edits, we automate the process by adopting an off-the-shhe the-shelf hand-object detector [49] to identify frames with hand-object contact, from which we randomly select one for editing. See Supp. C.4 for a detailed analysis. For greater applicability self-supervised manner, using the original object
and fiexibility, each stage is trained separately image as I Consequently, the stage-l model not only plays a crucial role in the video editing pipeline but also serves standalone image editing tool. The full pipeline is illustrated in Figure 3.
3.2 Stage I: HOI-aware Object Swapping in One Frame
In the first stage, given a source frame Z, a binary mask M; (where indicates the object's region and 0 denotes the background) and reference object image / the objective is to generate an edited frame It where the reference object naturally blended into the source frame. We aim for the generated content I₁ to exhibit HOI awareness and spatial alignment with I, ensuring that the

---

optial eaanuppll
VAC Encoder &
Diffusion Proseess
Danossing UNet
mrakk
Loss
VAC Encoder &
Diffision
atlention
Eneoder S
Denoising UNet
select L.
collage
VAl Encoder &
Hol
Impoint hand
Object sugmen tarion
DINO Enooder
warp
closs
Stage
Stage
I
X
Two-stage Inference Pipeline
refierence
stage
II
object
opeical flew extraction
warped
yawarg
generated
Figure 3: HOI-Swap involves two stages, each trained separately self-supervised manner. In stage I, an image diffusion model Ep, is trained to inpaint the masked object region with strongly augmented version of the original object image. In stage II, one frame selected from the video to serve as the anchor. The remaining video is then warped using this anchor frame, several points sampled within it, and optical flow extracted from the video. video diffusion model cØ, trained to reconstruct the full video sequence from the warped sequence. During inference, the stage-I model swaps the object in one frame. This edited frame then serves as the anchor for warping new video sequence, which is subsequently taken input for the stage-Il model to generate the complete video.
reference object is realistically interacting with human hands and accurately positioned within the scene context of the source frame.
Constructing paired training data Training conducted in a self-supervised manner. Below, we detail how prepare pseudo data pairs for training.
Masked frame preparation: For each frame I,, we obtain the object's bounding box (denoted by Mbbhoz from its segmentation mask M,, To prevent the bounding box shape from influencing the aspect ratio of the generated results, we adjust the bounding box be square. The masked frame thus derived by applying the square bounding box mask to the frame, yielding I,"h I, Mbboz We then crop and center the original frame to focus on the object region, enhancing its visibility.
Reference object preparation: We extract the object image I I; O (1 -M;),, which may be incomplete or partially obscured due contact with hands or other objects. To address these occlusions, we employ an off-the-shelf text-guided inpainting model [40] to fill in the missing parts of the object, using the object's name as the text prompt.
Object augmentation: Directly forming training pair (I,ah) II I2) suboptimal as it does not reflect the variability expected in real-world applications. At test time, the reference object may appear in any pose, orientation, or size. To bridge this gap between training and testing scenarios, apply strong augmentation techniques: (1) Spatially, we enhance the object's diversity by applying random resizing, flips, rotations, and perspective transformations: (2) Temporally, instead of using the object image from its original frame i, we introduce variability by randomly selecting alternate frame, Trabj_ from all available frames in the source video that contain the object. Finally, for preservation of the reference object's ' identity and structural details, we collage the augmented reference object image I onto the masked region in I.", producing I,"
We design the stage-I model as an image-conditioned latent diffusion model (LDM). Specifically, employ a pretrained variational autoencoder (VAE) [27] & used in prior work [45] to encode a frame Z into a latent space representation. This encoding is denoted by = EZZ), where I E H 2 E 4/ H/8 H and W denote the frame's 's height and width, respectively. During the forward diffusion process, Gaussian noise gradually added to over T steps. producing sequence of noisy samples {20, ... 2t, }T}. J. We train denoising network es, (o, t) that learns reverse the diffusion process, which implemented as a UNet [46]. The training objective of stage
Model design
5

---

I is summarized as:
Lstagel E2 e dobi (0..)) [|]e - EØ, (2,, dd .
(1)
Our LDM is designed take in two types of conditional signals alongside the standard 2, and (1) The reference object image I E RR is encoded through DINO [3] for distinctive object features dohj E RR765, dobi is then taken by co, as input to guide the denoising process via cross-altention mechanisms; (2) The masked frame ITh E RR is encoded by the same VAE to produce 2"h = E(.)). 2nh E 84////8 is then concatenated channel- with z before being fed into E0,-
3.3 Stage II: Controllable Motion-guided Video Generation
In stage II, given the first-stage edit Z,, source video V, and its binary mask sequence M, the objective is generate a new video V that propagates the single-frame edit across the remaining frames. For this purpose, we perform warping to transfer pixel points from the edited frame to the remaining frames, resulting in sparse and incomplete video sequence. We then train video diffusion network that learns to correct and fill in the gaps, completing the video editing process.
Constructing paired training data Training continues in self-supervised manner. For this purpose, we use the original frame I₁ from V during training and only replace I4 with the stage-I edit Z during inference time. The detailed process is explained below:
Masked frame sequence preparation. We employ the same frame masking strategy in stage and apply it to a sequence of frames V = {I. To standardize the masking across the sequence, we identify the largest bounding box bbbor from the sequence {Mo \N and mask each frame
by ITrh I, @ Maheer resulting in a masked frame sequence {I.T) M fi=1 1-=1- This ensures that the model is trained to inpaint a consistent object region across different frames.
max
Conditioning frame selection. To avoid limiting the model to generating videos based on specific reference frame, such as the first frame as often used in existing image-to-video generative models, we randomly select a frame I. from V as the conditioning frame. This mny-frame-conditoon mechanism brings additional flexibility during inference (for detailed discussion, see Supp. C.4).
Controllable motion guidance Given that object swapping can result in changes to shape or functionality, our approach is designed to flexible and adaptable, allowing for varying degrees of motion pattern encoding from the source video. The key idea is to control the level of motion agreement with the source via points sampled within the masked region. Tracking a large number of points over time captures extensive motion information from the source video, but it also reveals much about the source object's characteristics (e.g. shape). Conversely, using few points reduces the motion information and object characteristics carried over, but offers the model more freedom to generate plausible motion for the target. The latter scenario is particularly useful when only partial motion transfer is desired due to differences between objects, exemplified in Figure 2 (c).
To be specific, our approach involves the following steps: (1) uniformly sample r% pixel points within the original object's region in the conditioning frame I. (i.e., where Mbbos equals 1), (2) track these points using optical flow vectors computed by RAFT [52], and (3) warp a new video sequence based on these tracked points, using I. as the anchor. The resulting warped sequence is denoted by ywarp = [I LN The anchor frame Iwarp is intact and serves as the basis for warping (i.c., Twarp- = I.). For the other frames (i ل c), we overlay these points tracked from the conditioning frame e to frame onto the masked region of I, (i.e., I."), ), creating a composite image I, See Figure (upper right) for illustration of this process.
We vary the sparsity level of sampled motion points, from 0 to 100 during training, preparing the model for range of scenarios- from no motion guidance to full motion information. During inference, we demonstrate that the model capable of generating video sequences with varying levels of motion alignment to the original video (see Section 4.2 and Supp. C.4 for further discussion). Thus, our stage-II training goal is to take the warped yet incomplete video sequence (yworp) as input, then learn to fill in the gaps and smooth over discontinuities to reconstruct the full video (V).
Model design We design the stage-Il model as a video LDM. The LDM architecture and training objective closely follow those of stage I, with some adaptations for handling video sequences. We continue to use the VAE encoder E, but now apply it to sequence of frames V E x

---

to obtain the latent feature Z E R BXX4X1/88x/75 We inflate the 2D UNet into a 3D UNet as the denoising network (denoted by Eg2 (o,t)) for generating a frame sequence. Specifically, we insert additional temporal layers into the 2D UNet and interleave them with spatial layers, following [1, 10]. The stage-Il II training objective is shown below:
Estagel1 = Eg d' (--,(..)), [ (Zz, z" ())]]
(2)
Our video LDM involves two conditional signals in addition to the standard z, (a noised version of 2) and diffusion time step (: (1) The warped video sequence Vwarp E RN x3x xW is processed similarly as V. encoded by E, producing a latent feature z"" E IR X//xX// z" is then concate- nated channel-wise with z before being fed into the denoising network EØg. (2) The selected frame Ie besides serving as the anchor in the warping process, is also encoded by CLIP [43], yielding d² R that provides high-level contextual guidance and is incorporated in cØ, via cross-attention. Additionally, please see Supp. C.4 for an ablation of comparing CLIP and DINO as the object encoder.
4 Experiments
To comprehensively evaluate HOI-Swap, we consider both the image editing (accomplished by the stage-I model) and video editing (facilitated by the entire pipeline) task. We describe the experimental setup in Section 4.1 and present editing results along with an ablation analysis in Section 4.2.
4.1 Experimental Setup
Datasets Our training leverages two large-scale egocentric datasets, HOI4D [35] and Ego- Exo4D [15], which feature abundant HOIs, making them particularly suitable for exploring this problem. total, the stage-I model trained on 106.7K frames and the stage-ll model uses 26.8K 2-second video clips for training. The evaluation set for image editing includes 1,250 source images, each paired with four reference object images, resulting in a total of 5,000 edited images. The evalua- tion set for video editing is composed of 25 source videos, also combined with four object images each, yielding 100 unique edited videos. Note that: (1) For image editing evaluation, we include both scenarios, which challenge the model's HOI-aware capabilities, and non-hand present scenarios (20%) to evaluate its general bbjepi-swapping capability. (2) For video editing evaluation, in addition to HO14D videos, we include challenging in-the-wil videos from third-person videos from TCN Pouring [18]-dataset not used in training to assess the model's s zero-shot generalization across diverse scenarios. See Supp. B.1 for complete dataset descriptions.
-on-haad--resent
[8] and
Baselines For image editing, our stage-I model is compared with two strong image inpainting approaches, PBE [57] and AnyDoor [7], along with a hand-insertion approach Afford Diff [62] in the HOI domain. For video editing, we consider the following baselines: (1) applying the best image editing baseline to each frame of the video; (2) integrating the best image editing baseline with Vny [28], a recent video editing approach that takes one edited frame from any black-box image editing model as conditional guidance; (3) VideoSwap [17], the state-of-the-ar approach in customized object swapping. Besides these abject-image-guus guided approaches. we include a focused comparison with text-guided approaches for a thorough evaluation. See Supp. C.6 for the discussion.
Evaluation HOI-Swap is evaluated through both human and automatic metrics. We conduct a user study, involving 15 participants that assess 260 image edits and 100 video edits. Participants
Table Quantitative evaluation of HOI-Swap using (1) automatic metrics: contact agreement (cont. agr.), hand agreement (hand agr.), hand fidelity (hand fid.), subject consistency (subj. cons.), and motion smoothness (mot. smth.) the last two are for video only and (2) user preference rate (user pref.) from our user study. For video editing, users were given an option to report if they found all edits unsatisfactory, which has an average selection rate 10.9%. All values are percentages.
Image Editing
cont.
hand
hand fid.
user pref.
Video Editing
subj.
mot.
cont.
hand
hand fid.
user
agr.
agr.
cons.
smth.
agr.
agr.
pref.
PBE [57]
67.8
71.9
73.9
4.5
Per-frame
87.4
95.3
86.4
64.8
82.2
0.2
AnyDoor [7]
84.6
70.9
86.3
15.6
Any [28]
71.7
95.6
68.3
30.8
27.0
1.2
Afford Diff [62]
78.6
19.5
92.9
7.6
VideoSwap [17]
91.8
97.8
87.9
52.6
81.9
1.2
HOI-Swap (ours)
87.9
79.8
97.4
72.1
HOI-Swap (ours)
92.4
98.2
93.1
78.6
97.6
86.4
7

---

Source
AnyDoor
Frame
Sourco
Pur-Fratee
Any-2
Viche
HCI Swwpp
Figure 4: Qualitative results of HOI-Swap. We compare HOI-Swap with image (left) and video editing approaches (right). The reference object image is shown in the upper left corner of the source image. For image editing, HOI-Swap demonstrates the ability to seamlessly swap in-contact objects with HOI awareness, even in cluttered scenes. For video editing, HOI-Swap effectively propagates the one-frame edit across the entire video sequence while accurately following the source video's s motion, achieving the highest overall quality among all methods. We highly encourage readers to check Supp. C.1 and the project page video for more comparisons.
are presented with four edited results, three from baselines and one from HOI-Swap (randomly shuffled) and asked pick the best one in terms of object identity, HOI realism, and overall quality (for image edits), motion alignment and overall quality (for video edits). Each sample is assessed by three different participants. For automatic evaluation, we employ several metrics across different dimensions: (I) HOI contact agreement and hand agreement to measure spatial alignment, following [62, 66]; (2) hand fidelity, following [37]; (3) subject consistency and motion smoothness from VBench [23] to evaluate general video quality. See Supp. B.3 for full descriptions.
The stage-I model is trained on 512x512 resolution images for 25K steps. For stage-II training, input video resolution is set as 14x256x256. where we sample 14 frames at an fps of 7 and train the model for 50K steps. See Supp. B.4 for full details.
Implementation
4.2 Editing Results
Quantitative evaluation Table compares the performance of HOI-Swap on both image and video
editing tasks. For image editing, the two inpainting approaches, PBE [57] and AnyDoor [7] struggle to generate the hand correctly, as indicated by hand fidelity scores below 90%. Afford Diff' [62] is developed for the HOI context and also benefits from HOI-specific training data HOI4D. produces high-fidelin hands but does not address the editing task adequately, resulting in low contact and hand agreement with the source image. Our proposed HOI-Swap excels in both aspects, achieving superior performance in agreement and fidelity scores. In the human evaluation, HOI-Swap is consistently favored over the other three approaches, achieving a preference rate of 72.1%. For video editing, HOI-Swap outperforms all competing methods across all metrics, including general video quality metrics (subject consistency and motion smoothness) and HOI edit metrics (contact agreement, hand agreement and hand fidelity). It significantly surpasses the three baseline approaches, winning the user study with a selection rate of 86.4%. Recognizing the inherent challenges of the video editing task, we included survey question asking users if AAl edits are of very low quality: none successfully swap the object." " This response was selected at an average rate of 10.9%. See Supp. C.3 for details.
Qualitative evaluation Figure 4 shows HOI-Swap's edited images (left) and videos (right) alongside baseline comparisons. For image editing, HO1-Swap exhibits strong HOl-awareness, accurately

---

warped sequence 1awer
video
Source
HO 101-3300 penneneee videoss
plintssss
Figure 5: Ablation study on sampled motion points, comparing no to full motion points sampling. Left: we visualize ywarp, used as conditional guidance for the stage-ll II model. Note that row displays ywarp based on the source frame and is for illustration only, not provided to the model. Right: HOI-Swap exhibits controllable motion alignment: with sampled points, the generated video diverges from the source video's s motion; with full motion points, it closely mimics the source.
adjusting hand-holding patterns based on the reference object's properties (rows and 2, with source images from HO14D [35]). Moreover, HOI-Swap serves general object-swapping tool (not limited to HOIs) and can seamlessly swap in-contact objects in cluttered scenes (rows 3 and 4, with source images from EgoExo4D [15]). The final row demonstrates HOI-Swap's zero-shot generalization capability, using source images from EPIC-Kitchens [8]. Despite the out- of-distribution samples, HOI-Swap successfully aligns the reference mug handle with the source's hand context, whereas baselines like AnyDoor [7] fail to establish this connection. See Supp. C. for more examples.
For video editing (Figure right), the source video depicts hand rotating a toy The per-frame approach struggles with temporal consistency of the object identity and produces unnatural HOI motions. Any [28] fails to propagate the first-frame edit through subsequent frames, leading to inconsistencies. VideoSwap [17] preserves the overall look of the source video but fails to swap the reference object (i.e., the green toy car). In contrast, HOI-Swap delivers video edits that realistically depict the HOI and faithfully follow the source video's motion. Beyond this specific example, we note that baseline methods generally perform poorly on our in-contact object swapping task across various examples (see the project page video for more results). Despite our best efforts to reproduce their results using the official code and adjusting hyperparameters according to the provided guidelines, the performance of these models remained notably suboptimal. We observe similar issues with state-of fh-thh--rt text-guided approaches (Supp. C.6). This persistent underperformance highlights a big gap in current (otherwise quite powerful) diffusion models in their ability to accurately model the intricacies of HOIs, indicating great opportunities for further advancements this field.
Ablation study on motion points sparsity HOI-Swap can generate videos with different degrees of motion alignment with the original video. Figure 5 compares the generated results when sampling no points versus full points. Here the source video depicts a closing trash can lid action. row when no points are sampled (resulting in warping), the generated video does not follow the closing lid motion but instead depicts an action of putting the lid into the bin; visible hand motion outside the inpainted region provides context, guiding the model to generate a plausible sequence with the hand moving downward. Conversely, using all motion points produces videos that closely align with the source video's 's motion. See Supp. C.4 for follow-up discussion on this.
One-stage vs. two-stage design Next we verify the efficacy of our two-stage design, where we train a one-stage model variant that takes the reference object image and the masked frame sequence as input and outputs the edited video. Figure 6 shows both qualitative and quantitative comparisons. The one-stage approach does not yield satisfactory results, specifically failing to preserve the reference object's identity. This limitation arises from the challenge of simultaneously addressing both spatial and temporal alignment, thereby reinforcing the advantage of our two-stage design.

---

$4.7%
Source
Chaodaaee huadox Tho-stige B0I.5
Tha
12.2%
One- stage
Nasaa
Two stage
91.3%
Figure 6: Qualitative and quantitative comparisons between a one-stage baseline [1] and our two-stage HOI-Swap. The one-stage model struggles with preserving the new object's identify and fails to generate accurate interaction patterns, yielding inferior quantitative performance.
Moreover, we present a comprehensive discussion on various design choices of HOI-Swap in Supp. C.4 and C.5, covering aspects such object encoder choice, object masking strategy, editing frame selection strategy, and discussion on sampling regions.
5 Limitations and Future work
The problem of swapping in-contact objects in videos poses significant challenges. While HOI- Swap marks an important first step towards addressing these complexities, we recognize its current limitations. Specifically, we identify three areas for improvement: (1) generalization to new objects. HOI-Swap capably swaps new object instances with HOI awareness (e.g., swap an unseen mug with differently-shaie bowl). More challenging scenarios involve presenting very different unseen reference object images for swapping. For example, the model needs to accurately depict a hand holding scissors, though scissors were never part of the training dataset. This scenario requires the model to equip "world knowledge" that extends beyond its training data, enabling it to understand and realistically model how hands interact with broader variety of objects. (2) generalization to long video sequences with complex HOL. The two-stage pipeline of HOI-Swap is motivated by the observation that HOI interactions remain stable throughout a short video clip. For instance, for picking up mug sequence, replacing the mug with bowl can be reliably done in one single frame and propagated across the remaining frames. However, for longer sequences, it is possible that an object undergoes multiple distinct hand interactions that change dynamically over time. This complexity necessitates the development of methods that can capture and model these varied HOI interactions across time. (3) controllability. Our proposed controllable motion guidance allows HOI-Swap to freely choose the degree of motion alignment with the source video in editing. One future work direction is to enhance this controllability with spatial support, allowing the model to specify which regions of the source video should be targeted for motion transfer. See Supp. D for more details on HOI-Swap's limitations and failure modes, including visual examples.
6 Conclusion
Recognizing the limitations of current diffusion models in effectively capturing HOIs, we introduce HOI-Swap, novel approach for swapping hand-interacting objects in videos based on one reference object image. HOI-Swap consists of two stages: the first seamlessly integrates the reference object into a single frame, while the second propagates this edit across the entire sequence, with controllable motion alignment. Both qualitative and quantitative evaluations demonstrate that HOI-Swap produces realistic video edits with accurate HOIs that align well with the source content. In all, this work broadens the capabilities of generative models in video editing and represents the initial step towards solutions that can adeptly handle the intricacies of complex and dynamic HOIs in videos.
10

---

Acknowledgements:: UT Austin is supported in part by the IFML NSF AI Institute. KG is paid as research scientist at Meta. The authors would like to thank Zhengqi Gao (MIT), Xixi Hu (UT Austin), Hanwen liang (UT Austin), Feng Liang (UT Austin), and Yue Zhao (UT Austin) for their insightful discussions and valuable feedback, which contributed the development of this work.
References
[1] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22563 2201632-2555--- 2023.
[2] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages (650-9660, 2021.
[4] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23206-23217, 2023.
[5] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consisteney- aware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 20050, 2023.
[6] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-videdd Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840. 2023.
[7] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: sero-shot object-level image customization. arXiv preprint 2:Xiv:2307.09488, 2023.
[8] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference computer vision (ECCV), pages 720-736, 2018
[9] Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction genera- tion. arXiv preprint 160097 2023.
[10] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346-7356, 2023.
[11] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar arXiv:2208.0/4 0/6188 2022.
[12] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373, 2023.
[13] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv.2311.10709 2023.
[14] Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang. and Humphrey Shi. Pair-diffusion: Object- level image editing with structure-and and-aapearaaaeee paired diffusion models. arXiv preprint arXiv:2303.1 /754, 2023.
[15] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyl- los Afouras, Kumar Ahuutsh,, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.18259, 2023.
11

---

[16] Jing Gu, Yilin Wang. Nanxuan Zhao. Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang. HyunJoon Jung, and Xin Eric Wang. Swapanything Enabling arbitrary object swapping in personalized visual editing. arXiv preprint arXiv: 240..05777 2024.
[17] Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang. Videoswap: Customized video subject swapping with interactive semantic point correspondence. arXiv preprint arXiv:2312.00085 2023.
[18] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- -rh Prompt-to- to-p-oppt image editing with cross attention control. arXiv preprint 2Xiv:2208.01626. 2022.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[20] Hezhen Hu, Weilun Wang, Wengang Zhou, and Houqiang Li. Hand-object interaction image generation. Advances in Neural Information Processing Systems, 35:23805-23817, 2022.
[21] Zhihao Hu and Dong Xu. Videocontrolnet: A muotion-guided video-to-video translation frame- work by using diffusion model with controlnet. arXiv preprint arXiv:2307.14077 2023.
[22] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Com- poser: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778, 2023.
[23] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.
2023.
[24] Inbar Saaberman-SSiegegglas Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. arXiv preprint arXiv:2304.061400 2023.
[25] Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns. Dall-e-bot: Introducing web-scale diffusion models to robotics. IEEE Robotics and Automation Letters, 2023.
[26] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[27] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 1312.61 14, 2013.
[28] Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen, Anyv2v: A plug-and-play framework for any video-to-video editing tasks. arXiv preprint arXiv:2403. 14468, 2024.
[29] Sumith Kulal, Tim Brooks, Alex Aiken, Jiajun Wu, Jimei Yang, Jingwan Lu, Alexei A Efros, and Krishna Kumar Singh. Putting people in their place: Affordance aware human insertion into scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17089-17099, 2023.
[30] Pika Labs. https://pika..rt,, 2024.
[31] Bolin Lai, Xiaoliang Dai, Lawrence Chen, Guan Pang, James M Rehg, and Miao Liu. Lego: Learning egocentric action frame generation via visual instruction tuning. arXiv preprint arXiv:2312.038849 2023.
[32] Yao-Chih Lee, Ji-Ze Genevieve Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang. Shape- aware text-driven layered video editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14317-14326, 2023.
[33] Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9801-9810 10, 2023.
[34] Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, al. Flowvid: Taming imperfect optical flows for consistent vidco-to-vides synthesis. arXiv preprint arXiv:2312. 17681, 2023.
12

---

[35] Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang. Zhoujie Fu, He Wang. and Li Yi. Hoi4d: 4d egocentric dataset for category-leve human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21013-21022, 2022.
[36] Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. Tf-icon: Diffusion- training-fres cross-domain image composition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2294-2305, 2023.
[37] Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. Put myself in your shoes: Lifting the egocentric perspective from exocentric videos. arXiv preprint MrXiv:2403.06351,, 2024.
[38] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108 01073, 2021.
[39] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296-4304 2024.
[40] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lly Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2/ 10741, ,0771,22211 2021.
[41] Elia Peruzzo, Vidit Goel, Dejia Xu, Xingqian Xu, Yifan Jiang, Zhangyang Wang, Humphrey Shi, and Nicu Sebe. Vase: Object-centric appearance and shape manipulation of real videos. arXiv preprint arXiv:2401.02473, 2024.
[42] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15992-15942, 2023.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 PMLR, 2021.
[44] Nikhila Ravi, Valentin Gabeur, Yuan-Time Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma. Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007 14, 2024.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted Intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[47] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 02500-22510 10, 2023.
[48] Pierre Sermanet, Corey Lynch, Jasmine Hsu, and Sergey Levine. Time-contrastive networks: Self-supervised learning from multi-view observation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 486-487. IEEE, 2017.
[49] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9869 9878, 2020.
[50] Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, and Daniel Aliaga. Objectstitch: Object compositing with diffusion model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18310-18319,
2023.
13

---

[51] Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, He Zhang, Wei Xiong, and Daniel Aliaga. Imprint: Generative object compositing by learning preseityp representation. arXiv preprint arXiv:2403. 10701, 2024.
[52] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
[53] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. houg-and-plas diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1921-1930, 2023.
[54] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024.
[55] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video; One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023.
[56] Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models. arXiv preprint arXiv:2310. 10647, 2023.
[57] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18381-18391, 2023.
[58] Mengjiao Yang. Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint 14X1v:2310.06114 2023.
[59] Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In SIGGRAPH Asia 2023 Conference Papers, pages 1-11, 2023.
[60] Danah Yatim. Rafail Fridman, Omer Bar Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. arXiv preprint arXiv:2311. 17009, 2023.
[61] Yufei Ye, Poorvi Hebbar, Abhinav Gupta, and Shubham Tulsiani. Diffusion-guided reconstruc- tion of everyday hand-object interaction clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19717-19728, 2023.
[62] Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song. Shubham Tulsiani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
pages
[63] David Junhao Zhang, Dongxu Li, Hung Le, Mike Zheng Shou, Caiming Xiong, and Doyen Sa- hoo. Moonshot: Towards controllable video generation and editing with multimodal conditions. arXiv preprint arXiv:2401 240.00827.. 2024.
[64] Hui Zhang, Sammy Christen, Zicong Fan, Otmar Hilliges, and Jie Song. Graspxl: Generating grasping motions for diverse objects at scale. arXiv preprint arXiv:2403. 194.9999..1949, 2024.
[65] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[66] Menggi Zhang, Yang Fu, Zheng Ding, Sifci Liu, Zhuowen Tu, and Xiaolong Wang. Hoidiffusion: Generating realistic 3d hand-object interaction data. arXiv preprint arXiv:2403. 12011, 2024.
[67] Zhichao Zuo, Zhao Zhang, Yan Luo, Yang Zhao, Haijun Zhang, Yi Yang, and Meng Wang. Cut- and-paste: Subject-driven video editing with attention control. arXiv preprint 1 2023.
14

---

Contents
Supplementary Video
15
Experimental Setup
15
B.1 Datasets
15
B.2 Baselines
16
B.3 Evaluation
16
B.4 Implementation
18
C Editing Results
18
C.I More qualitative results
18
C.2 Detailed quantitative results
18
C.3 User study analysis
21
C.4 Ablation study
C.5 Discussion on sampling region
24
C.6 Comparison with text-guided editing approaches
25
D Limitations
26
Societal Impact
26
A Supplementary Video
We invite readers to view the Supp. video available at https://vision: utexas edu/ projects/ Socts/HOI-Swapp which presents additional qualitative results of HOI-Swap. The video show- cases video editing comparisons with baselines approaches, demonstrates zero-shot generalization on EPIC-Kitchens and TCN Pouring datasets (on which the model was not trained), includes an ablation study on varying the number of sampled motion points, and presents failure cases HOI-Swap.
B Experimental Setup
B.1 Datasets
We adopt all available videos in HOI4D [35] due to its comprehensive per-frame object mask annotations. We use 2,679 videos for training and hold out 292 videos for evaluation; the evaluation videos are selected based on object instance ids ensure that the source objects during test time unseen by the model. In total, the dataset offers 79.9K frames (sampled at the original 15 fps) and 26.8K 2-second video clips for training. HOI4D features object categories, ranging from rigid objects like bowls, mugs, and kettles to articulated objects like laptops, pliers, and buckets. For evaluation, we select 1,000 images and 17 vidcos from the hold-out set as the source and combine each source with four randomly chosen reference object images.
For EgoExo4D [15], utilize the ground truth object segmentation masks available at I fps for stage-I training. Since our video model operates at 7 fps. a future enhancement could involve interpolating these masks to scale up stage-II training with EgoExo4D videos and the corresponding pseudo masks. We focus on 18 frequently occurring object categories where editing is relevant: bowl, container, bottle, package, knife, pan, spoon, plate, spatula, pot, mug, cup, jar, glass, jug, kettle, and skillet. We adopt frames belonging to these categories and follow the official split, resulting in 18.7K training frames. For evaluation, we take 250 images from the official validation split as sources and combine each with four randomly chosen reference object images, resulting in 1,000 edited images.
15

---

HOI-Swap Image Human Evaluation Survey
Notes to read before you start:
Image Editing Tack Definiation Given orgginal frame and mage of reference object, Image Editing aims seap the object from the original frame with the reforenes object seamlesly, maintanning the consistency frame contest and overall appearance of reference object.
choices for the quesions,
baselires produce all all-blck images, Don't appear
coused failure load images.
uncumfertabl: appeccisic your paliatee. image segmentatio issees, ressiting good image edit? An example (please link slides).
bocline may pournic
objects appear
imogularitiss
Questions for you to answer.
Original
object
iragu provose idinty okjon
Q2: Which demonaratzs the hea handwbicct intoration? Yiu reed sdlect chokez. You shoald creikentt following erizrias crallation: (1) flw realistic the hand- otjac vnsrection (2) how will the the efterd image allee with the velemmee onjer's ourontic (hand greping woeld vey fora mag and myy carl and original frome' come hord contert?
B
highoes guilty? dbject Mecniiy
qualey" How realleti hand objec intenctiool (3) well
fie referesue objec's iemantte (hand grasping would vay fix
viginal nore's scone and castont
0
Figure 7: Human evaluation interface for image editing part. We provide a source frame for editing alongside image of the reference object. Users asked to evaluate and select their favorite edited results based on various image editing criteria.
Additionally, we test the model's zero-shot editing capabilities on four EPIC-Kitchen [8] videos, which depict in-the-wild kitchen scenarios with often cluttered backgrounds, and four videos from TCN Pouring [48], with two being egocentric and two exocentric, featuring pouring motions. This evaluation encompasses variety challenging scenarios thoroughly assess model performance.
B.2 Baselines
We implement the baselines described Section 4.1 using their official code. Baselines including PBE, AnyDoor and V2V adopted in zero-shot manner. Afford Diff utilizes the same training data source (HOI4D) as ours. We do not provide additional layout or hand orientation information to Afford Diff. VideoSwap [17], one-shot-tuned video editing approach, is trained the source video using the default setup in their official repository. AnyDoor is recognized as the best image editing method and integrated into the two video editing baselines, per-frame and V2y
Note that these baseline methods impose more demanding test-time requirements compared to ours. PBE and AnyDoor both necessitate precise segmentation masks of the source object, while HOI- Swap only requires bounding box during inference. In addition, VideoSwap requires users to click semantic points for every video, provide text prompt describing the video, and train specialized object model using a few reference images. This extensive prepossessing limits its scalability and is a contributing factor to the smaller size of our video evaluation set. In contrast, HOI-Swap is fast inference, with RAFT optical flow extraction completing in just a few seconds per vidco, making efficient and practical for real-world applications.
B.3 Evaluation
Automatic evaluation For automatic evaluation, we employ several metrics across different dimen- sions: (1) To measure how well the generated content aligns spatially with the source image, use HOI contact agreement and hand agreement. Following [62, 66], we utilize the in-contact state label provided by off-the-sheii hand detector [49]. The HOI contact agreement score is determined
16

---

HOI-Swap Video Human Evaluation Survey
Notes to read hefore you start:
Video loading tike some time.. Please ensure the video has fully leaded before making your selections..
Video Edilding Tisk Definiation Given origiaal vidco and an image of reforesce objec,, Vidco Editiing aims the object from the original video with the referene object seamlesly, maintaining the coméstency seene contest and overall appearance of referenee objoct.
In the chuisce for questions.
Some baseline modkis graseals hends that sppear unzatun incomioraalee We appecciate potimnor. unnatuaa due
Some reference objects
segmentation resultingg visble holes irreglarities.
What is good video edit? An example (please click the link access the slides). good alignment? An exumple (please
access slides).
Questions you to
Original video
Ql: mution aligamen? A good malon aligumeet mxae acecmbal same untion
losdld One Oaeccinioo.
soop
cvatnation:
Interntivn (3) Now well loss the motion geneuted video sligh with the glarl video?
120 foow well
with the ebject's semantic original vidre's hand contest?
@di the fillowing suserncat tue or fibe? 'Al odis are vau low qually, aore socessfllly soup the objar. I caade a choiee is the secoad qustiz ouly bosase was requined to."
The File O
Figure 8: Human evaluation interface for video editing part. We provide a source video for editing alongside image of the reference object. Users asked to evaluate and select their favorite edited videos based on various video editing criteria.
by checking if the in-contact label of the generated frame matches the source frame. Additionally, we calculate mIOU between the generated and source hand regions using the detector's bounding boxes for hand agreement. (2) To evaluate how realistic the generated hand is, follow [37] and assess hand fidelity based on the hand detector's confidence score. (3) To evaluate video quality, we consider subject consistency (DINO feature similarity across frames) and motion smoothness (de- termined by motion priors from a video frame interpolation model [33]) as proposed in VBench [23]. Note that we exclude object agreement from the automatic evaluation due to its inaccuracies. Often, the generated hand occludes some object regions, leading to lower similarity scores compared to simply placing the reference object an area with no or partial hand presence. This discrepancy does not accurately reflect HOI realism and generation quality. To ensure comprehensive evaluation, we incorporate questions in our user study address this dimension.
study with 15 participants who assessed 260 image edits and 100 video edits randomly selected from the evaluation set. The human evaluation survey interfaces for image and video editing shown in Figure 7 and Figure 8, respectively. Each survey began with general guidelines on the evaluation process, followed by examples of high-quality image or video editing results and good motion alignment for videos. For each question, participants were presented with four edited results: three from baselines and one from HOI-Swap, randomly shuffled. Each image or video sample was evaluated by three different participants minimize bias.
Human evaluation We conducted
For each image editing question, participants were asked to select the best one or two edits based on the following three evaluation dimensions:
1. Object Identity: The ability to preserve the identity of the reference object.
2. HOI Realism:
The realism of the hand-object interaction and how well the edited image aligns with the reference object's semanties (e.g., hand grasping varies for mug and a toy car) and the original frame's scene and hand context.
3. Overall Quality: The seamlessness of the object swap, considering:
17

---

Does the generated object maintain the identity of the reference object image?
How realistic is the hand-object interaction?
How well does the edited image align with the reference object's semantics and the original frame's s scene and hand context?
For each video editing question, participants were asked to select the best one two edits based on the following two evaluation dimensions:
1. Motion Alignment: The extent to which the edit matches the motion of the source video.
2. Overall Quality:
The seamlessness of the object swap, considering:
Does the hand-object interaction look realistic?
How well does the edited video align with the reference object's 'sject's semantics and the original video's scene and hand context?
How well does the motion in the generated video align with the original video?
Recognizing the inherent challenges of the video editing task, we included a sanity check sub-question asking users if "All edits are of very low quality; none successfully swap the object."
B.4 Implementation
For data preprocessing, we apply several transformations the reference object image: random rotations with angles ranging from -90 to 90 degrees, random perspective changes, and random horizontal flips. We crop and center the source frame using variable ratio between 0.3 and 0.6, determined during training to introduce variability; this ratio is defined as the size of the bounding box relative to the image size. For stage-I training: image resolution set as 512x512 We train the model for total of 25K steps with learning rate of le-4 and batch size of 8. We finetune the entire 2D UNet for image editing. For stage-Il training: input video resolution is set as 14 x56 2566, where we sample 14 frames at fps of 7. The model is trained for total of 50K steps with learning rate of le-5 and a batch size of We finetune the temporal layers of the 3D UNet. A classifier-free guidance dropout rate of 0.2 is employed for all stages. Training for each stage takes about 3 days on one ANVIDIA-VI0--1 GPU node. We set sampling points sparsity value as 50% for all quantitative evaluations.
C Editing Results
C.1 More qualitative results
Beyond the examples shown in Figure of the main paper, we provide additional qualitative image editing results of HOI-Swap in Figures 9 and 10. HOI-Swap excels in both structured scenarios with relatively clean backgrounds (Figure source frames from HOI4D and TCN Pouring), and challenging in-the-wild scenarios featuring cluttered backgrounds (Figure 10, source frames from EgoExo4D and EPIC Kitchens). Compared to the baselines. HOI-Swap seamlessly integrates the reference object into the target scene, demonstrating exceptional HOI awareness and the ability to accurately reorient the reference object to align with the scene context. Moreover, it also performs well in general object-swappinn settings where no hand is involved, as shown in rows 3, 5, and 6 of Figure 10. Please refer to our project page for more qualitative results of video editing.
C.2 Detailed quantitative results
Table I of the main paper reports video editing results when videos are split based on object instances. In addition, we conduct experiments with two splitting ways: (1) across different subjects, and (2) across different actions. The results are reported Table 2. HOI-Swap consistently outperforms baseline approaches across different data splitting settings, demonstrating its effectiveness.
Furthermore, we provide a breakdown of video editing results Table 3, separating results from in-domain videos (HO14D) and out-of-domain videos (EPIC-Kitchens & TCN Pouring). We observe that the performance gain of HOI-Swap higher for in-domain videos than out-of-domain videos.
Future work could explore extending HOI-Swap's generalization capabilities.
18

---

Source
Ref Object
PBE
AnyDoor
Afford Diff
HOI-Swap
Region to be inpainted
Figure 9: Qualitative image editing results of HOI-Swap on structured scenarios, with source images from HOI4D and TCN Pouring.
19

---

Source
Ref Object
PBE
AnyDoor
Afford Diff
HOI-Swap
Region inpainted
Qualitative image editing results of HOI-Swap on challenging in-the-will scenarios, with source images from EgoExo4D and EPIC-Kitchens.
Figure 10:
20

---

Table 2:
Video editing results with splitting by subject and action (Table in the main paper reports
results with splitting by object instances).
split by subject
split by action
Method
subj.
mot.
cont.
hand
hand fid.
subj.
mot.
cont.
hand
hand
smth.
agr.
agr.
smth.
agr.
agr.
fid.
Per-frame [8]
85.6
95.2
73.6
63.3
62.9
84.5
94.7
82.8
57.6
71.0
Any [29]
67.2
95.3
68.2
27.7
10.0
69.8
94.7
74.6
30.6
19.9
VideoSwap [18]
88.0
97.4
71.8
29.6
69.0
87.2
97.1
87.7
39.8
79.4
HOI-Swap (ours)
90.0
98.2
80.4
84.4
90.3
89.6
97.8
93.3
77.4
94.6
Table Video editing results breakdown: in-domain videos (left) and out-of-domain videos (right).
In-domain videos
Out-of-domain videos
Method
subj. cons.
mot. smth.
cont.
hand
hand fid.
subj. cons.
mot. smth.
hand agr.
hand fid.
user
agr.
agr.
pref.
agr.
pref.
Per-frame [8]
85.8
94.7
81.3
61.5
78.4
0.3
91.0
96.4
97.3
71.9
90.2
0.0
V2y [29]
73.1
95.5
70.9
33.2
27.0
1.7
68.8
95.7
62.7
25.6
26.9
0.0
VidcoSwap [18]
90.5
97.5
82.3
38.9
74.8
0.4
94.4
98.1
99.6
81.5
97.1
2.5
HOI-Swap (ours)
91.3
98.0
89.9
79.0
96.6
82.7
94.7
98.6
99.8
77.8
99.5
70.2
C.3
User study analysis
Table 4 provides the breakdown of the evaluation dimensions for our HOI-Swap vs. baseline models on both image and video editing tasks. In terms of all metrics for both image and video editing, our HOI-Swap's 's editing results are consistently preferred by humans. Note that an exception identity, where HOI-Swap does not surpass Afford Diff. This is because Afford Diff focuses on synthesizing hands on fixed objects, directly copying and pasting the object image from the given reference object image.
object
Table 4: Breakdown of user study evaluation metrics. For image editing, we consider metrics including object identity, HOI realism, and overall quality. For video editing, consider metrics including motion alignment and overall quality. Each value represents the frequency (expressed as a percentage) with which participants prefer that model considering particular evaluation dimension. Standard deviations are calculated across participants to illustrate preference variability. For video editing, users given additional option indicate if they found all edits unsatisfactory, which has an average rate of 10.9%. Note that our HOI-Swap does not surpass Afford Diff in the ability to preserve object identity. This is because Afford Diff focuses synthesizing hands fixed objects so the model will directly copy and paste the object image from the given reference object image.
object identity
HOI realism
overall quality
motion alignment
overall quality
Image Editing
Video Editing
PBE [57]
2.0
11.47.11
4.5 = 2.1
Per-frame
0.2 + 0.8
0.2 = 0.8
AnyDoor [7]
40.3 11.1
12.7 7.3
15.6 6.5
A2V [28]
1.6 2.8
1.2 2.7
Afford Diff' [62]
86.2 + 13.2
5.1 4.3
7.6 = 6.8
VideoSwap [17]
2.8 3.4
1.2 2.0
HOI-Swap (ours)
70.1 + 6.8
71.5 - 9.1
72.1 = 10.8
HOI-Swap (ours)
84.5 + 10.3
86.4 8.6
C.4
Ablation study
DINO vs. CLIP encoder Stage and stage II of our pipeline employ the DINO and CLIP encoders, respectively, to align with their specific objectives. Stage I, focused on swapping the reference object within a single frame, benefits from the DINO encoder due to its enhanced ability to capture "objectness" compared with the CLIP encoder. The main emphasis stage II is to transfer motion from the source video, and CLIP encoder is adopted provide scene context for generating the video. We additionally conduct experiments to explore the possibility using a DINO encoder for stage II. As reported in Table 5, the two encoder variants perform similarly.
Ground truth vs. estimated object masks While it generally assumed that users will provide the ground truth bounding box segmentation mask of the object they wish to replace during
21

---

Frame
Frame 4
Frame 7
Frame 10
Frame 13
Source
HOI Swap generated videos
Edited Framne Region to be inpainted
Figure 11:
We compare generated results using different frames as the stage-I edit frame. Selecting the HOI contact frame (frame 7 in this example) for editing yields the best results.
Table 5:
Video editing results: comparison of DINO and CLIP encoders for stage II. The two object encoders yield similar performance.
subject consistency
motion smoothness
contact agreement
hand agreement
hand fidelity
Video Editing
HOI-Swap (DINO encoder)
91.2
98.1
88.9
79.9
96.2
HOI-SSap (CLIP encoder)
91.4
98.0
89.9
79.0
96.6
inference [57, 7], we acknowledge the potential of incorporating automatic segmentation methods to reduce user effort. Following this, we applied SAM-2 [44] to identify bounding boxes on test videos as an alternative to manually providing ground truth. This method requires just single click inside the object in the initial frame, followed by SAM-2 automatically tracking the target object. The quantitative comparisons are presented in Table 6. While there is some degradation across three HOI metrics, we believe this feature is valuable for downstream applications as it greatly eases the input requirements and improves user convenience. Note that the baseline approaches have more stringent input requirements than HOI-Swap, e.g. need precise object segmentation masks additional text prompts. Even with the use automatically generated masks, HOI-Swap demonstrates its great advantages over the baselines.
Table 6: Video editing results: comparison of using ground truth object bounding boxes vs. SAM-2 estimated model input.
subject consistency
motion smoothness
contact agreement
hand agreement
hand fidelity
Video Editing
Prior best
90.5
97.5
82.4
61.5
78.4
HOI-Swap (SAM-2 bbox)
91.2
98.0
87.8
73.9
91.8
HOI-Swap (GT bbox)
91.4
98.0
89.9
79.0
96.6
Masking strategy In stage we mask the source frame with square bounding box, as opposed to using the source object's segmentation mask in [57]. This masking strategy not only directs
22

---

Sourco
Reference
Sourre (maskkad)
Generated image
Sourbe (massedd)
image
Comparison of two masking strategies. The stage-I model using segmentation mask (row only learns to fill the missing region without HOI awareness. In contrast, square bounding box (row 3) also masks the hand-object interaction regions, requiring the model to reconstruct these interactions; this enables our stage-I model to effectively learn diverse HOI patterns from data.
Figure 12:
the model to fill the predefined masked area but also to generate plausible hand-object interactions aligning with the source frame's hand position and the reference object. To demonstrate this point, we experimented with a variant of the stage model that uses the original object's segmentation mask instead. As illustrated in Figure 12, this variant struggles with grasp changes when the reference and source objects differ, thereby reinforcing the effectiveness of our chosen approach.
Editing frame selection One question naturally arising from two-stage editing pipeline is determining which frame from the video sequence to edit. Recall that our training strategy (Sec- tion 3.3) prepares the stage-II model be any-frame frame-conditioned bringing maximum inference flexibility. Here, our key insight is to choose a hand-object contact frame for editing due to two main advantages: (1) Hand-object contact frames particularly challenging as they require generating correct interaction patterns (e.g. grasps) compared with non-contact frames; prioritizing them ensures that the model handle the most complex scenarios. (2) Hand-object contact frames offer the most comprehensive scene context for the stage-1 model, aiding in spatial alignment.
To illustrate, in Figure 11, compare video editing results when choosing the 1st, 4th and 7th frames in sequence of picking up a mug. First, evident that the 7th frame is the most challenging to edit since it requires precise handling of the grasp region, while the 1st and 4th frame involve no or little hand interaction. Next, while single-frame edits for the 1st and 4th frame look reasonable (successfully swapping the mug), the lack of hand context in these frames leads to incorrect orientation of the mug handle. Consequently, this causes failures or imperfections in generating the subsequent frames 10 and 13, where the hand is unrealistically holding the mug.
Motion points sparsity Following the discussion in Section 4.2 of the main paper, we present comprehensive analysis of the impact of sampled point sparsity. Figure 13 illustrates two swapping cases, involving the motion of closing trash can lid. For the left video, the two trash cans differ greatly in shape and closing mechanism. Using all points confuses the model, resulting in an incorrect lid-closing motion. Conversely, using no points generates different motion of putting down the trash can rather than closing the lid. In this example, using 50% of the points yields the most plausible edit. For the right video, as discussed the main paper, due to the similarities between the two objects, replicating the full motion by adopting 100% points is effective. In all, HOI-Swap allows users flexibly decide the extent of motion transfer when generating content, accommodating different scenarios and object characteristics.
23

---

0% pointta
0%
50% points
50%
100% poires
100%
Figure 13: Ablation study of sampled motion points sparsity. The left figure illustrates a scenario where only partial motion transfer is desired, due to differences between the original and new object. The right figure showcases scenario where full motion transfer beneficial, owing to the similarities between the objects. We invite readers to view these examples in project page.
C,5 Discussion on sampling region
As described in Section 3.3, we perform uniform sampling of points within the bounding box region. When the new object differs greatly in shape from the original, sampled points might incorrectly capture motion background points could be assigned foreground motion, and vice versa. A direct solution mitigate this mismatch is reduce the sampling density. Nevertheless, HOI-Swap demonstrates considerable robustness even when large number of points are sampled, adeptly managing potential discrepancies within the warped sequence Ywarp, As shown Figure 14, with 50% points sampled, the visualized ywarp reflects the original object's 's shape to certain extent, yet the model effectively distinguishes between background and the foreground object, recognizing the object as whole and thereby generating plausible videos.
find that
We identify two potential reasons for the model's effectiveness. First, our training strategy helps enhance the model's ' capability to discern between foreground and background elements, since we intentionally sample points within the bounding box region rather than solely within the object's segmentation mask. Consequently. the model is trained with mix of in-object points and background points, preparing it to handle background points that may sampled outside the new object region during inference. Additionally, the relative constancy of background and clear distinction between foreground and background contributes the model'ss robustness, allowing it to recognize the object whole and accurately move it even when some regions inaccurately carry motion.

---

Sourcoe
Sauras
Wirpped Stat.
Warped San.
Gerannted Viduo
Gereitnnc Vidko
Figure 14: HOI-Swap demonstrates remarkable robustness, when object shapes differ greatly, leading
motion mismatches of sampled points. Even with large sampled points density (50%),
effectively handles the flawed warped sequences yearp and produces high-quality video edits.
Source
Pika
P199
1562
Runway
Rerender
HOI HOI-SWar Swap
Figure 15: Comparison of HOI-Swap with text-guided diffusion models: Pika [30], Runway [10], and Rerender-a-vided (Rerender) [59]. To evaluate these latest models the object swapping task, we describe the reference object in text and prompt the models to replace the original object in the video. These approaches are unable to alter the shape of the bowl and fail to swap the original bowl with kettle as required.
C.6 Comparison with text-guided editing approaches
To understand how the latest text-guided diffusion models perform on this task, we provide qualitative comparison of HOI-Swap with three models: Pika [30], Runway [10], and Rerender A Video [59]. We adopt the text prompt: "change the object in the video to kettle while keeping background and
25

---

Smurror
Wirpped Shrt
Warped
Generatec Video
Cereentte Vicho
Figure 16: Failure cases of HOI-Swap. Left: stage-I failure, Right: stage-ll failure.
motion the same", which proved most effective compared to complex prompts. As shown in Figure 15, these approaches edit the video in way that preserves the original object shape, failing to generate the new object adapt the HOI patterns. These results underscore the unique capabilities of HOI-Swap.
D Limitations
One area for improvement stems from the sampling region discussion in Section C.5, with sampled points potentially carrying wrong motion when object shape changes, particularly when a large number of points are sampled. To further refine our model's performance, future direction to explore fine-grained spatial control in motion alignment. Specifically, concentrated sampling could provide the model with the precise ability to modulate specific regions of an object. These enhancements are anticipated to augment the model's 's adaptability and accuracy in editing complex HOI sequences.
Another direction the development of an automated method to determine the sparsity of motion points based the object swapping scenario and the motion complexity in the original video. This would allow the model to autonomously decide the extent of motion transfer necessary when generating new content.
addition, we present failure cases Figure 16. The left example shows a stage-I failure, where HOI-Swap fails achieve accurate spatial alignment. The right example illustrates stage-II failure, where noise in the warped sequence impedes model's understanding of the reference object, leading to wrong motion patterns.
Finally, we emphasize that the problem of swapping in-contact objects in videos poses significant challenges, particularly with complex HOI sequences. We recognize the limitations of HOI-Swap in its current stage and view this work as an initial step towards resolving these challenges. Looking ahead, we aim to incorporate more precise spatial and motion controls, and expand HOI-Swap's capabilities to handle longer video sequences with more intricate HOIs.
Societal Impact
This project aims equip the community with powerful tool for swapping customized objects into videos, particularly improving the realism hand-object interactions. However, there is a potential risk that malicious real-world individuals, potentially misleading viewers. This issue is not exclusive to our approach but is common concern across various video editing and generating modeling techniques. To mitigate such risks. one possible solution is to introduce subtle noise perturbations to published images disrupt the customization process. Furthermore, implementing invisible watermarking in the generated videos could help prevent misuse and ensure proper attribution of the content.
could misuse this technology to produce deceptive videos involving
26

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly describe our main claims such as the findings, contributions, and results in the abstract and introduction. They match our idea, methodology, and experimental results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We describe limitations of HOI-Swap in Supp. Section and provide its failure cases in Supp. video.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
27

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe the editing pipeline of HOI-Edit in detail in Section 3. Also, please see main paper Section 4.1 and Supp. Section for its full implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Please refer to our project website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss cttpssccccpbbcccccceeeeccecccccc /tt.g/eeee/eeeeeeeisieeeeee /Cooeeeooiissseeeeee cassionPoocyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please see main paper Section 4.1 and Supp. Section B for full experimental setup of HOI-Swap.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see Table 4 for error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see main paper Section 4.1 and Supp. Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tee///nerrrree cc htte//ullcc/eee [ csCuidel ines?
Answer: [Yes]
Justification:
We strictly conform the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed the potential negative societal impact of this work when being used to create misleading content in Supp. Section E. We also discussed possible solutions to mitigate the misuse.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
The focus of our paper is on video editing, specifically swapping in- in-contact objects in video sequences. This application does involve the generation of misleading information, nor does it use datasets or models prone to misuse (we train our models using HOI4D and EgoExo4D data).
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please see main paper Section and Supp. Section
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "Mac0En8uK5", "reviewer": "Reviewer_ih2o", "summary": "The work considers the task of swapping the objects in ego-centric short clips with hand-object interactions. The manuscript claims to introduce this sub-task within the field of generative video editing. Concretely, HOI-Swap starts from RGB video, object area bounding box, and an image of the target object, and generates a new video, transferring hand movement to the new object, possibly adapting to the changing functionality of the object. The method has two stages: inpainting a single frame, and extending it to the whole sequence ensuring consistency. HOI-Swap is thoroughly evaluated against recent video and image editing methods, also the work demonstrates qualitative examples of generalization to novel datasets.", "questions": "1) Stages 1 and 2 employ different image encoders (DINO and CLIP), what is the motivation for this choice?\n2) Lines 212-220 discuss the ability to vary the number of control points sampled for motion guidance. What is the default value used for quantitative evaluations?", "limitations": "A detailed discussion of limitations is present in the Appendix with complementary examples in a supplementary video.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The HOI-Swap method is novel and addresses a yet unexplored problem;\n2) The proposed method is technically sound. It employs several techniques to ensure robustness and generalization of the resulting model: reference object augmentation technique to ensure robustness to viewpoint orientation, robust masking (square mask, consistent across frames to prevent overfitting), and randomized selection of the anchor frame.\n3) Another notable feature of the method is its controllability through motion guidance. This original idea enables to variation of the amount of preserved information from the source object's motion to the target, thus allowing the method to adapt to the changing functionality of the objects.\n4) The manuscript is well-written and easy to follow.", "weaknesses": "A minor weak point is the lack of detailed discussion on downstream applications of the model. The Introduction section touches on this topic, however, an example of possible demos or a deeper discussion of possible usages of the model could help to better motivate the importance of the introduced task and the proposed method. \n\nAnother possible area for improvement is automatic input mask annotation. Currently, the method requires a bounding box of the object as an input, however, with current advancements in the segmentation methods this requirement can be alleviated (or at least quantitatively compared to the GT bbox usage).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ih2o for the helpful comments and for providing thoughtful feedback on our work.\n\n---\n\n**1. Downstream applications**\n\n***(i) Entertainment.*** As showcased in Figure 1 of the main paper and Supp. video (pages 2-3), HOI-Swap can be applied in scenarios where object modification is required without reshooting footage. For example, in advertising, there may be situations where a pre-recorded video needs to adapt to new sponsorship requirements by replacing a soda can in the video with a water bottle. HOI-Swap offers a practical tool to seamlessly swap the objects, while maintaining realistic hand interaction patterns that adapt to the new object's shape and affordance.\n\n***(ii) Data augmentation in robotics.*** There is an increasing trend of teaching robots through videos of humans performing tasks (Ln 24-26). By recording just a single video of picking up a mug, HOI-Swap can generate multiple variations of this video with different objects (e.g., bottles, bowls, kettles), all following the same motion trajectory. This capability can greatly reduce the need for extensive data collection in robotics. \n\n---\n\n**2. Automatic input mask annotation**\n\n> Currently, the method requires a bounding box of the object as an input, however, with current advancements in the segmentation methods this requirement can be alleviated (or at least quantitatively compared to the GT bbox usage).\n\nWe appreciate the suggestion. While it is generally assumed that users will provide the ground truth bounding box or segmentation mask of the object they wish to replace during inference [8, 57], we acknowledge the potential of incorporating automatic segmentation methods to reduce user effort. Following this, we applied the recently released SAM-2 to identify bounding boxes on test videos as an alternative to manually providing ground truth. This method requires just a single click inside the object in the initial frame, followed by SAM-2 automatically tracking the target object. The quantitative comparisons are presented in the table below. While there is some degradation across three HOI metrics, we believe this feature is valuable for downstream applications as it greatly eases the input requirements and improves user convenience. Note that the baseline approaches have more stringent input requirements than ours, e.g. need precise object segmentation masks, additional text prompts (see Ln 590-596). Even with the use of automatically generated masks, HOI-Swap demonstrates its great advantages over the baselines. We will add this discussion in the paper. \n\n| Model    | CLIP consistency | Motion smoothness | Contact agreement | Hand mIOU | Hand confidence |\n|-------------|:----------------:|:-----------------:|:-----------------:|:---------:|:---------------:|\n| Prior best\t                    | 90.5             | 97.5              | 82.4              | 61.5      | 78.4            |\n| HOI-Swap (SAM-2 bbox)    | 91.2             | 98.0              | 87.8              | 73.9      | 91.8            |\n| HOI-Swap (GT bbox)          | 91.4             | 98.0              | 89.9              | 79.0      | 96.6            |\n\n---\n\n**3. Motivation of DINO vs. CLIP encoders**\n\n> Stages 1 and 2 employ different image encoders (DINO and CLIP), what is the motivation for this choice?\n\nPlease refer to our general response.\n\n---\n\n**4. Default sampling points sparsity for motion guidance**\n\nThe default value used for quantitative evaluations is 50%. We appreciate your pointing it out and will include this information in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ko0epNm9wy", "reviewer": "Reviewer_NtRs", "summary": "This article focuses on proposing a two-stage network HOI-Swap, a video editing framework designed for precise object edits with HOI awareness.To address the problem of real perception of HOIs as well as spatial and temporal alignment with the original video, HOI- Swap first stage focuses on solving HOI awareness and establishing spatial alignment by training an image restoration diffusion model to swap objects in one frame. The second phase deforms video sequences from edited frames by tracking inter-frame points with randomly sampled optical flow, and trains a video diffusion model to generate new videos.", "questions": "1.line 174, this paper suggests that data augmentation techniques such as flipping, rotating, perspective transformations, etc. are performed due to the difference in size and pose of the target and reference objects, so why not use scaling techniques to bridge the gap between large and small objects?\n2.Line205, this paper proposes to apply the largest bounding box as mask to the whole video sequence, so for the case of a relatively large range of object movement, will a larger mask range lead to the proportion of sampling points that does not truly reflect the proportion of extracted object features and motion information？(For example, if the target object only occupies 1/4 size of the mask in a certain frame, then a uniform sampling ratio of 50% is also difficult to extract the object motion information).\n3. Why not continue the use of DINO for the first stage network instead of switching to CLIP in the second stage network?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well organized and easy to understand.\n2. The ablation experiment fully confirms the effectiveness and contribution of the improvement measures proposed in this paper.", "weaknesses": "1. Limited innovation. The methodology of this paper is a combination of existing work and is more like a technical report than an academic paper.\n2. Lack of dynamic HOI awareness transfer. The second stage does not embody HOI interaction awareness. The single-frame propagation only propagates the motion information of the object and does not embody how the HOI-aware information is conveyed in the video sequence.\n3. Insufficient experiments. This paper only demonstrates HOI awareness in single-frame image restoration, and in the video aspect, either the scene with little change in hand poses or the scene with little change in object characteristics is demonstrated, but for the complex video sequences with large changes in object characteristics and the need for dynamic changes in hand pose, the paper does not give the relevant effects to demonstrate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer NtRs for the helpful comments and for providing thoughtful feedback on our work.\n\n---\n\n**1. Limited innovation**\n\n> The methodology of this paper is a combination of existing work\n\nWe respectfully disagree. As acknowledged by three other reviewers, our paper presents both a novel task and approach. Reviewer ih2o noted that “The HOI-Swap method is novel and addresses a yet unexplored problem”, reviewer gbKB mentioned it “presents a novel approach”, and reviewer rdgi recognized it as addressing “a novel task: HOI video editing”.\n\n***Clarifications of innovations.*** In the related works section, we thoroughly discuss existing approaches and outline their differences with us. Our experimental results (Ln 316-321) further demonstrate that the leading generative methods fall short in producing satisfactory edits within HOI scenarios, thus underlining the importance and necessity of HOI-Swap.\n\n***Innovation highlights.*** (i) Problem Space: We investigate HOI-aware object swapping in videos, an area where current leading generative models falter (see Figure 4 of the main paper). This problem space has not been previously explored, and our work serves to fill this gap in the field. (ii) Technical Contributions: Our proposed editing framework simplifies complexity by dividing the task as two stages. The first stage introduces HOI awareness and spatial alignment—elements notably missing in existing image editing methods. The second stage offers controllable motion alignment with the original video, utilizing motion points and warping techniques; this approach stands in stark contrast to existing video editing approaches that enforce 100% motion alignment with shape-based conditional signals.\n\nWe would appreciate it if reviewer NtRs could identify any potentially overlooked references, allowing us to further clarify and set apart our contributions.\n\n---\n\n**2. Clarification on HOI awareness**\n\n> The second stage does not embody HOI interaction awareness. The single-frame propagation only propagates the motion information of the object and does not embody how the HOI-aware information is conveyed in the video sequence.\n\nWe appreciate your comment and clarify below. HOI-awareness is a spatial property that can be reliably captured in a static frame and generally remains consistent over time.  In our study, we observe that across the four datasets we use, HOI interactions are stable throughout the sequence. As an intuitive example, when swapping a hand-held mug, one HOI frame suffices to conceptualize how a new object would replace the original, including any necessary adjustments in hand grasp patterns; we then propage this edit to the remaining frames using our controllable motion guidance techniques. \n\nOur two-stage design is driven by this principle. We recognize the reviewer's concern regarding scenarios where an object may undergo multiple distinct hand interactions within a short clip, though these are uncommon (based on our observation across 4 datasets). As noted in Ln 741-745, we view our work as an initial step towards HOI-aware video editing challenges, and plan to explore more complex video sequences in future work. \n\n---\n\n**3. Complex video sequences with large object changes**\n\n> Either the scene with little change in hand poses or the scene with little change in object characteristics is demonstrated, but for the complex video sequences with large changes in object characteristics and the need for dynamic changes in hand pose, the paper does not give the relevant effects to demonstrate\n\nThrough various qualitative generation results in our paper and Supp. video, we show that HOI-Swap can adeptly address diverse object swaps in videos. For instance, Figure 1 of the main paper illustrates the replacement of a kettle with a bottle and a bowl—objects that greatly differ in shape and appearance from the original kettle. Figure R2 of the rebuttal PDF (first and last row) provides further evidence of its editing capabilities for differently-shaped objects. \n\nMoreover, our Supp. video includes scenarios featuring various object and action variations. Object variation examples include swapping a bottle with a mug (page 5), a bowl with a kettle (page 5) and a trash can with another differently-shaped one (page 9-10). For action variations, examples include closing a laptop display (page 4), picking up a scissor (page 4), uprighting a tilted bottle (page 5), pushing and rotating toy car (page 6), and closing trash can lid (page 9-10). \n\nWe invite the reviewer to provide more details on what they consider “complex video sequences with large object changes,” so we can address this aspect more thoroughly. Lastly, as noted in Ln 316-321, HOI-aware video editing is a very challenging problem, and our experiments reveal that even leading video generation approaches struggle with the scenarios we have tested. We acknowledge in our limitations statement (Ln 744-745) that improving HOI-Swap’s capabilities to handle longer video sequences with more intricate HOIs is an important future direction. \n\n---\n\n**4. Clarification on scaling**\n\n> Why not use scaling techniques…?\n\nWe indeed incorporate scaling as our data augmentation by randomly resizing the reference object (Ln 644-646 of Supp.). We'll ensure to clearly mention this in Ln 174 of the main paper.\n\n---\n\n**5. Clarification on bounding box**\n\nThe largest bounding box is used only for preparing the masked input sequence (Ln 205) to prevent exposing the source object to the model, while sampling points sparsity is decided by the conditioning frame’s bounding box (Ln 222). We'll clarify that these are separate processes in the main paper.\n\n---\n\n**6. Motivation of DINO vs. CLIP encoders**\n\nPlease refer to our general response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vApEuQ8Ocu", "reviewer": "Reviewer_gbKB", "summary": "This work presents a novel approach for object insertion in hand-object interaction scenarios. The proposed approach consists of two stages: image-based editing to for precise alignment of hand with the inserted object, and video-based editing for motion alignment with the original video. The model is trained in a self-supervised manner and also imparts a varying level of controllability to adjust the degree of motion alignment based on object changes. Extensive experiments on HOI4D, EgoExo4D, EPIC-Kitchens and TCN Pouring datasets show the effectiveness of the proposed approach in various scenarios.", "questions": "I need clarifications on the following aspects to better understand the results (more details in the weaknesses above):\n- For evaluation on HOI4D & EgoExo4D, how is the test split created? Do the held-out videos contain different object categories or different instances from the same category or different subjects performing the experiments or different actions being performed?\n- Do the results in Tab.1 span all 4 datasets? It'd be helpful to have a breakdown of the results in different settings to understand where the proposed approach is most effective.\n- Are all the baselines trained in the same setup as the proposed approach? It'd be useful to clearly describe the protocols followed for each baseline. Are these baselines retrained or used in a zero-shot manner?", "limitations": "L330-332 in the main paper defers the limitations and failure modes to the supplementary without providing any insights. Please add these details to the main paper.\n\n---\nI have read all the reviews and the rebuttal. I thank the authors for providing additional clarifications, which help me understand the paper better and I have increased my score to Weak Accept.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed 2-stage approach to decompose the complexity of the task is intuitive and handles the challenges of hand-object interaction changes, correct object placement and temporal alignment of HOI motion effectively.\n- The first stage can also be used in a stand-alone manner for image editing tasks and the controllability in motion alignment is a useful feature.\n- The self-supervised training strategy circumvents the need for collecting paired training data for this task..\n- Extensive experiments (Tab.1) on HOI4D & EgoExo4D show the effectiveness of the proposed approach over several baselines for both image and video editing tasks. The zero-shot generalization setting to EPIC-Kitchens and TCN Pouring datasets are also considered.\n- Visualizations in Fig.4,5 are helpful in understanding the capabilities of the proposed approach.", "weaknesses": "- Several details about the experimental setup are missing from the main paper. These details are important to understand the scope of the claims in the paper. While the supplementary contains more details, it'd be helpful to have more clarity on the following aspects:\n    - For evaluation on HOI4D & EgoExo4D, how is the test split created? Do the held-out videos contain different object categories or different instances from the same category or different subjects performing the experiments or different actions being performed? Having quantitative experiments in these different settings would help in understanding the benefits and limitations of the proposed approach.\n    - Do the results in Tab.1 span all 4 datasets? Again, it'd be helpful to have a breakdown of the results in different settings to understand where the proposed approach is most effective.\n- Are all the baselines trained in the same setup as the proposed approach? I understand that it might not be feasible to implement/train all baselines due to differences in architecture and compute requirements, but it'd be useful to clearly describe the protocols followed for each baseline. Are these baselines retrained or used in a zero-shot manner? For example, AffordDiff [62] only takes a single RGB image of the object as input and generates the HOI image, it does not take the reference video into account so it is expected that the object orientation may not be consistent with the video (L43-50). AffordDiff also allows for controllability in the layout & orientation of the generated hand so it can be modified to match the hand in the reference video. I'm not asking to reimplement AffordDiff to adapt to this new setting, but it'd be useful to have these details in the main paper to understand the differences between baselines and the proposed approach.\n- While the 2-stage decomposition of the approach makes sense, it'd also be useful to quantitively verify that it is indeed better than the 1-stage analog. For example, in Fig. 3, Stage-2 can take the DINO encoding of the reference object instead of the generated image from Stage-1 to create a 1-stage approach. This is not required for rebuttal, but a suggestion to validate the 2-stage approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer gbKB for the helpful comments and for providing thoughtful feedback on our work.\n\n---\n\n**1. Clarification on experimental setup**\n\n***(i) Data split***\n\nAs we focus on the problem of “object” editing, videos are split based on object instances (Ln 567-568). The held-out videos feature different instances of the same object categories, testing the model’s ability to swap unseen instances (e.g., a different mug not seen during training) rather than entirely new categories (e.g., scissors, if scissors were not included in the training data). Addressing object category-level generalization presents a significantly more challenging problem, requiring the model to generate plausible interactions with a completely unseen object category (e.g., accurately depicting a hand holding scissors, despite the model never encountering scissors during training). We acknowledge this as an exciting direction for future work.\n\nInspired by your comment on other data splitting ways, we conducted experiments with two new splits: (1) across different subjects, and (2) across different actions. The results are reported in Table R1 in the rebuttal PDF. HOI-Swap consistently outperforms baseline approaches in these new settings, demonstrating its effectiveness. \n\n***(ii) Baselines implementation***\n\nBaselines including PBE [57], AnyDoor [8] and AnyV2V [29] are adopted in a zero-shot manner. Afford Diff [62] utilizes the same training data source (HOI4D) as ours. We do not provide additional layout or hand orientation information to Afford Diff. VideoSwap [18], a one-shot-tuned video editing approach, is trained on the source video using the default setup in their official repository. \n\nNote that we also evaluate on two out-of-domain datasets (EPIC-Kitchens and TCN pouring), where all approaches (except VideoSwap which is tuned on the source video), including ours, are applied in a zero-shot manner. \n\nWe appreciate your feedback and will ensure these clarifications are included in the main paper.\n\n***(iii) Results breakdown***\n\nAs suggested, in Table R2 of the rebuttal PDF, we provide a breakdown of video editing results, separating results from in-domain videos (HOI4D) and out-of-domain videos (EPIC-Kitchens & TCN Pouring). Note that the video editing results in Table 1 span three datasets (HOI4D, EPIC-Kitchens and TCN Pouring) with EgoExo4D excluded due to its absence of high-FPS mask annotations (see Ln 575-577). We observe that the performance gain of HOI-Swap is higher for in-domain videos than out-of-domain videos. We will update our manuscript to incorporate these discussions.\n\n---\n\n**2. Validation of the two-stage approach**\n\nWe appreciate the suggestion to verify the efficacy of our two-stage design compared to a one-stage counterpart (though the reviewer says it’s not necessary for rebuttal) and have conducted additional experiments to assess this. We train one VideoLDM [2] that takes the reference object image and the masked frame sequence as input and outputs the edited video; this serves as a trained baseline for a one-stage object swapping approach. \n\nWe provide both qualitative and quantitative comparisons in Figure R1 of the rebuttal PDF. The one-stage approach does not yield satisfactory results, specifically failing to preserve the reference object’s identity. We believe this deficiency stems from the model’s need to handle both spatial and temporal alignments simultaneously. We will add this analysis in the paper.\n\n---\n\n**3. Inclusion of limitations in main paper**\n\nWe will add a summary of the key limitations and failure modes (currently discussed more in Supp. and Supp. video) directly into the main text to ensure comprehensive accessibility of these details within the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lpYUWvRbsA", "reviewer": "Reviewer_rdgi", "summary": "This paper presents HOI-Swap, a diffusion-based video editing framework for object swap editing. HOI-Swap involves two stages. In the first stage, the authors train an image-editing model to swap object in one frame. In stage II, the authors first warp a video from the edited frame using the optical flow sampled from the source video. Then a video diffusion model is trained to reconstruct the whole video from the warped frame sequences. Through this two-stage approach, HOI-Swap successfully addresses three main challenges in HOI video editing: (a) HOI-aware capabilities, (b) spatial alignment of the object with the hand, and (c) temporal alignment with the source video. Experiments demonstrate that HOI-Swap outperforms both image-editing and video-editing baselines.", "questions": "1. How can the image-editing model in stage I perceive HOI? The authors should provide more explanation and maybe some more ablation studies to support their results.\n2. The whole pipeline seems applicable to general object-swap tasks. Have the authors tested the model on other objects?\n3. Are the backgrounds in the test set all seen by the model during training?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper address a novel task: HOI video editing.\n2. This paper pinpoints the three main challenges facing this task.\n3. The video-editing stage employs optical flow from the source video to achieve temporal alignment with the source video.\n4. HOI-Swap demonstrates better performance than existing video-editing approaches in HOI video editing.", "weaknesses": "1. Stage I does not seem to address HOI-awareness explicitly. The authors should explain why the image-editing model can perceive HOI.\n2. The authors did not mention the generalizability of their method across unseen kinds of objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer rdgi for the helpful comments and for providing thoughtful feedback on our work.\n\n---\n\n**1. HOI awareness in stage I**\n\n> How can the image-editing model in stage I perceive HOI?...maybe some more ablations\n\nThe model’s ability to perceive HOI fundamentally relies on the data it is trained on. By providing the model with a large volume of HOI images, it learns the diverse ways in which hands interact with various objects. HOI-rich data acts as the driving force, equipping the model with the necessary insights to accurately replicate and understand these interactions.\n\nIn terms of the specific techniques, in stage I, we mask the source frame with a square bounding box (Ln 162-166), as opposed to using the source object’s segmentation mask as in [57]. This masking strategy not only directs the model to fill the predefined masked area but also to generate plausible hand-object interactions aligning with the source frame’s hand position and the reference object. To demonstrate this point, we experimented with a variant of the stage I model that uses the original object's segmentation mask instead. As illustrated in Figure R2 of the rebuttal PDF, this variant struggles with grasp changes when the reference and source objects differ, thereby reinforcing the effectiveness of our chosen approach. We will incorporate this discussion in the paper.\n\n---\n\n**2. Applicability to general object swaps, testing on other objects?**\n\n​​Yes. As noted in Ln 567-568, we designed our train-test split based on object instances, to evaluate our model’s ability to generalize to unseen object instances, such as a new mug not encountered during training. Our qualitative results indicate that HOI-Swap can handle novel object instances and deliver good edits.  Moreover, as demonstrated in Figures 4 and 8 of the paper, HOI-Swap performs robustly in general object-swap tasks, in cluttered scenes where no hand is present, showcasing its strong capability across diverse swapping scenarios. \n\nOn the other hand, object category-level generalization requires the model to generate plausible interactions with a completely unseen object category (e.g., accurately depicting a hand holding scissors, despite the model never encountering scissors during training), is considerably more challenging and not validated here. This requires the model to acquire a broader understanding of novel interactions. We acknowledge this as an exciting direction for future work.\n\n> The whole pipeline seems applicable to general object-swap tasks. Have the authors tested the model on other objects?\n\nFollowing the discussion above, our experimental setup aims at assessing generalization across new object instances. If the reviewer could specify or elaborate on “other objects,” we would be happy to address this further.\n\n---\n\n**3. Background generalization**\n\n> Are the backgrounds in the test set all seen by the model during training?\n\nWe provide zero-shot video editing results on two datasets (EPIC-Kitchens and TCN Pouring), where the test videos feature backgrounds not encountered by the model during training. This showcases HOI-Swap’s generalization ability to new backgrounds.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful and constructive review of our manuscript. 

Three of the four reviewers recommend accepting. We were encouraged that the reviewers found our problem and approach novel (ih2o, gbKB, rdgi), technically sound (ih2o), and that our paper pinpoints and addresses this task’s challenges effectively (rdgi, gbKB), backed up with extensive experiments (gbKB), ablations (NtRs) and helpful visualizations (gbKB). 

In response to the feedback received, we provide a general response here to address a query shared by two reviewers (ih2o and NtRs), and individual responses below to specific points from each reviewer. Please refer to the attached rebuttal PDF, where supplementary figures and tables have been provided to substantiate our responses.

   
---

**Motivation of DINO vs. CLIP encoders**


We thank Reviewer ih2o and reviewer NtRs for raising this question. Stage I and stage II of our pipeline employ the DINO and CLIP encoders, respectively, to align with their specific objectives. Stage I, focused on swapping the reference object within a single frame, benefits from the DINO encoder due to its enhanced ability to capture “objectness” compared with the CLIP encoder. The main emphasis of stage II is to transfer motion from the source video, and a CLIP encoder is adopted to provide scene context for generating the video. Given the reviewers’ query, we additionally conduct experiments to explore the possibility of using a DINO encoder for stage II. As reported in the table below, both encoder variants perform similarly. We will add this discussion in the paper. 

|                          | CLIP consistency | Motion smoothness | Contact agreement | Hand mIOU | Hand confidence |
|:------------------------:|:----------------:|:-----------------:|:-----------------:|:---------:|:---------------:|
| HOI-Swap (DINO encoder)  |       91.2       |       98.1        |       88.9        |   79.9    |       96.2      |
| HOI-Swap (CLIP encoder)  |       91.4       |       98.0        |       89.9        |   79.0    |       96.6      |

---

We would again like to thank all reviewers for their time and feedback, and we hope that our responses adequately address all concerns. Any further questions are highly welcomed.

Meta-review: The paper studies the problem of swapping objects in hand-object interaction videos. The authors present a diffusion-based video editing framework. Comprehensive evaluations show that HOI-Swap outperforms existing methods, delivering high-quality video edits. Three reviewers think that their concerns are mostly addressed, and one reviewer further proposed that this paper should clarify its downstream scenarios further explain its capability boundaries, and raise the scores. After reading this paper, reviews, and responses, the AC agrees that this work made a good contribution.
Decision: Accept (poster)