Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos
Authors: Matthew Chang
Aditya Prakash
Saurabh Gupta
Venue: NeurIPS 2023 poster (2023)
Keywords: Inpainting
Diffusion
Robot Learning
Egocentric Vision
TL;DR: We develop an inpainting diffusion model to factorize egocentric videos into agent and environment representations, and show the power of the factored representation for downstream tasks in vision and robot learning.

Abstract: The analysis and use of egocentric videos for robotics tasks is made challenging by occlusion and the visual mismatch between the human hand and a robot end-effector. Past work views the human hand as a nuisance and removes it from the scene. However, the hand also provides a valuable signal for learning. In this work, we propose to extract a factored representation of the scene that separates the agent (human hand) and the environment. This alleviates both occlusion and mismatch while preserving the signal, thereby easing the design of models for downstream robotics tasks. At the heart of this factorization is our proposed Video Inpainting via Diffusion Model (VIDM) that leverages both a prior on real-world images (through a large-scale pre-trained diffusion model) and the appearance of the object in earlier frames of the video (through attention). Our experiments demonstrate the effectiveness of VIDM at improving the in-painting quality in egocentric videos and the power of our factored representation for numerous tasks: object detection, 3D reconstruction of manipulated objects, and learning of reward functions, policies, and affordances from videos.

Look Ma, No Hands! Agent-Environman Factorization of Egocentric Videos
Matthew Chang Aditya Prakash Saurabh Gupta University of Illinois, Uhbann-Champaign {mc48, adityap9, saurabig)@i1limois edu
Abstract
The analysis and use of egocentric videos for robotic tasks made challenging by occlusion due to the hand and the visual mismatch between the human hand and a robot end-effector. In this sense, the human hand presents a nuisance. However, often hands also provide valuable signal, e.g. the hand pose may suggest what kind of object being held. this work, we propose to extract a factored representation of the scene that separates the agent (human hand) and the environment. This alleviates both occlusion and mismatch while preserving the signal, thereby easing the design of models for downstream robotics tasks. At the heart of this factorization is our proposed Video Inpainting via Diffusion Model (VIDM) that leverages both prior real-world images (through a large-scale pre-trained diffusion model) and the appearance the object in earlier frames the video (through attention). Our experiments demonstrate the effectiveness of VIDM at improving inpainting quality on egocentric videos and the power our factored representation for numerous tasks: object detection, 3D reconstruction of manipulated objects, and learning of reward functions, policies, and affordances from videos.
Introduction
Observations of humans interacting with their environments, as in egocentric video datasets [12, hold the potential scale up robotic policy learning. Such videos offer the possibility of learning affordances [2, 19], reward functions [1] and object trajectories [57]. However, key bottleneck in these applications is the mismatch in the visual appearance the robot and human hand, and the occlusion caused by the hand.
Human hands can often be nuisance. They occlude objects of interaction and induce domain gap between the data available for learning (egocentric videos) and the data seen by the robot at execution time. Consequently, past work has focused on removing hands from the scene by masking [19] or inpainting [1]. However, hands also provide valuable signal for learning. The hand pose may reveal object affordances, and the approach of the hand toward objects define dense reward functions for learning policies.
In this work, we propose the use of factored agent and environment representation. The agent representation is obtained by segmenting out the hand, while the environment representation is obtained by inpainting the hand out of the image (Fig. 1). We argue that such a factored representation removes the nuisance, but at the same time preserves the signal for learning. Furthermore, the factorization allows independent manipulation of the representation necessary. E.g., the agent representation could be converted to a form that is agnostic to the embodiment for better transfer across agents. This enables applications in 2D/3D visual perception and robot learning (see Fig. 2).
Project website with code, video, and models:
gi github..o 1o/vidm.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Lagent
agent m
Agent Segmentation Model
Video Inpainting Diffusion Model (Section 4)
Agent- Environment Factored Representation
Figure 1: Agent-Environment Factorization of Egocentric Videos. Occlusion and the visual mismatch between humans and robots, make it difficult to use egocentric videos for robotic tasks. We propose a pixel-space factorization of egocentric videos into agent and environment representations (AEF, Sec. 3). An agent representation peeenr is obtained using a model to segment out the agent. The environment representation T. is obtained by inpainting out the agent from the original image using VIDM, a novel Video Inpainting Diffusion Model (Sec. 4). AEF enables many different visual perception and robotics tasks (Fig. and Sec. 5).
a) 2D/3D Visual Perception
b) A Affordances
Reward Functions
g
agent
I
gl 1 agent
I env
I env
1, env
Figure 2: Agent-Enviven Egent-Environment Factored (AEF) representations enable many applications. (a) For visual perception tasks (Sec. 5.2 and 5.3), the unoceluded environment Ien can be used in addition to the original image. (b) For affordance learning tasks (Sec. 5.4), the unoccluded environment Ie can be used to predict relevant desirable aspects of the agent in !. (c) For reward learning tasks (Sec. 5.5 and 5.6) agent representations can transformed into agent-agnostic formats for more effective transfer across embodiments.
But how do we obtain such factored representation from raw egocentric videos? While detection & segmentation of hands well-studied [13, 70, 103), our focus is inpainting. Here, rather than just relying on a generic prior over images, we observe that the past frames may already have revealed the true appearance of the scene occluded by the hand in the current time-step. We develop video inpainting model that leverages both these cues. We use large-scale pre-trained diffusion model for the former and an attention-based lookup of information from the past frames for the latter. We refer to it as Video Inpainting via Diffusion (VIDM). Our approach outperforms DLFormer [62], the previous Mate-offthh-an for video inpainting. by large margin (Tab. 1) and 8x faster at time.
Next, we demonstrate the ability of the factored representation across tasks spanning 2D/3D visual perception robot learning. Specifically, we adopt 5 existing benchmark tasks: a) 2D detection of objects interaction [13], b) 3D reconstruction of hand-held objects [59, 95], c) learning affordances (where interact and how) from egocentric videos [19], d) learning reward functions, and e) their use for interactive policy learning [1]. We show how selectively choosing and modifying aspects of the factored representation improves performance across all of these tasks compared to existing approaches. We believe the advances presented in this paper will enable the use of egocentric videos for learning policies for robots.
2 Related Work
Robot Learning from Hand-Object Interaction Data. Many past works have sought use human-object interaction data for robotic tasks. Researchers have used videos to predict: regions of interactions [19,50], grasps / hand pose afforded by objects [19,96, 96], and post- post-grasp trajectories [2,42].

---

E
A
C
Predicted Notse
Convolution
Ieput Images
vQ Encoder
Concat Noley
Donost
Block
and Mask
Multi-Frame Attention Block
Figure 3: Video Inpainting Diffusion Models (VIDM), We extend pre-trained single-frame inpaint- are introduced as
ing difffusion models [63] to videos. Features from context frames (4z-h, - (t-1) additional inputs into the Attention Block A. We repeat the multi-frame attention block 8 times (4 to encode and 4 to decode) to construct the U-Net [66] that conducts step of denoising. The U-Net operates in the VQ encoder latent space [63].
[59, 95] predict 3D shapes for hand-held objects. [1, 8, 69] learn reward functions from videos for learning real-world policies. Others use human-object interaction videos to pre-train feature representations for robotic policies [51, 60, 91]. While some papers ignore the gap between the hand and the robot end-effector [51, 60, 91], others adopt ways to be agnostic to the hand by masking out [19], inpainting [1], converting human pixels to robot pixels [73, 92], or learn embodiment invariant representations [8, 69, 100]. Instead, we pursue factored representation that allows downstream applications to choose how to use the agent and the environment representations. Different from [103], remove only the hand and retain the object of interaction.
Diffusion Models have been successful at unconditional [28, 74,777, and conditional [15,55,63] image synthesis for several applications, e.g. text-to- -mat generation [3,54,6] 61], image editing [27,43, 48], video generation 30, 86], video prediction and infilling [31] (similar to video inpainting, but focused on generating entire frames), 3D shape synthesis [45, 101, 106], self-driving applications [72, 107]. While diffusion models produce impressive results, they are computationally expensive due to the iterative nature of the generative process. To mitigate this limitation, several improvements have been proposed, e.g. distillation [44, 68], faster sampling [16, 75, 88), cascaded [29] and latent [63, 83] diffusion models. Among these. latent diffusion models (LDM) are the most commonly used variant. LDMs use an autoencoder to compress the high-dimensional image inputs to lower-dimension latent space and train diffusion models in this latent space. In our work, we build off LDMs.
Inpainting requires reasoning about the local and global structure of the image to perform image synthesis. While earlier approaches focus on preserving local structure [4, 11, 17, 23], they are unable capture complex geometries or large missing areas. Recent works alleviate these issues using neural networks [38, 40, 47, 58] to incorporate large receptive fields [46, 80], intermediate representations (e.g. edges [52, 93], segmentation maps [78]), adversarial learning [33,78, 97, 99, 104). With advancements in generative modeling, diffusion models have also been very effective on this task [43, 54, 63, 67, 96]. These approaches operate by learning expressive generative priors. An alternative is utilize videos (36,41,62,9)) 60] to reason about temporal consistency [32.53,87], leverage optical flow [79, 94], utilize neural radiance fields [81, 82), or extract context about the missing parts from the previous frames [35-37, 35-377877 87]. Recent video-based inpainting methods incorporate transformers [41, 62, 102], cycle-consistency [90] & discrete latent space [62] to achieve state-of-the- art results. While existing approaches inpaint the entire video, we focus inpainting specific frame (previous frames are used as context) as required by downstream tasks. Our work adopts diffusion models for video-based inpainting learn factored agent-environment representations, specifically, building on the single-frame inpainting model from [63].
3 Agent-Environment Factored (AEF) Representations
Motivated by the recent success of generative models, we develop our factorization directly in the pixel space. Given an image from egocentric video, factored representation decomposes

---

it into 1, and I!Th. Here, Th shows the environment without the agent, while 1, shows the agent (Fig. 1) without the environment. These two images together constitute our Agent-Environmen Factored (AEF) Representation.
Using the Factorization. This factorization enables the independent use of agent/ environment information as applicable. For example, when hands are source of nuisance (e.g. when object detectors are pre-trained on non-egocentric data without hands in Fig. 2a), we can use 4 in addition to 1 to get a better view of the scene. In other situations. it may be desirable to predict agent properties afforded by different parts of the environments (Fig. 2b). In such a situation, 1 can be used to predict the necessary aspects of 1 In yet other situations, while the location of the agent provides useful information, its appearance may be a nuisance (e.g. when learning dense reward functions from egocentric videos for finetuning policies for robots in Fig. 2c). In such a situation, models could be trained on _ and ((,temm)) where the function g generates the necessary abstractions of the agent image.
Extracting the Factorization from Egocentric Videos. For 1 Ta we employ segmentation models to produce masks for the agent in 1,. For egocentric videos, we use the state-of-the-art VISOR models [13] to segment out the hand. When the depicted agent a robot, we use DeepLab [9] pre- trained on MS-COCO [39] and fine-tuned using manually annotated robot end-effector frames. Robot end-effectors have a distinctive appearance and limited variety, hence we can train a high-performing model with only a small amount of training data. We denote the agent segmentation in I, by m; agent
Extracting T.thV is more challenging because of significant scene occlusion induced by the agent in given frame. Naively extracting m. by just masking out the agent leaves artifacts in the image. Thus, we inpaint the image to obtain 1.th. We condition this inpainting on the current frame 4, and previous frames from the video, i.e. {4g-h..... }-13 since occluded parts of the scene in 4, may actually be visible in earlier frames. This simplifies the inpainting model, which can steal pixels from earlier frames rather than only relying on generic generative image prior. We denote the inpainting m2a))
function as yl m ngeat {mi- ............ ........--111. J: (I--,..... Ie-1 }), and describe it next.
agent
Video Inpainting via Diffusion Models (VIDM)
The inpainting function mi agent "
{m, my - (lr-....., 4-11) inpaints the mask m agent in image 4, using information in images ,, through 4. The function p is realized through a neural model that uses attention [85] to extract information from the previous frames. We train this neural network using latent diffusion [63], which employs denoising diffusion [28] in the latent space obtained from a pre-trained vector quantization (VQ) encoder [18, 14].
agent
Model Architecture. We follow prior work [63] and train model co(2n, u. c), where 2, is the noisy version of the diffusion target z at diffusion timestep u, and is the conditioning (note we denote diffusion timesteps as u to avoid conflict with I, which refers to video timesteps). For each diffusion timestep u, this model predicts the noise e added to z. This prediction done in the latent space of the VQ encoder, and the difffusion model is implemented using a U-Net architecture [66] with interleaving attention layers [85] to incorporate conditioning on past frames. The use of attention enables learning of where to steal pixels from, eliminating the need for explicit tracking. Furthermore, this allows us to condition on previous frames while reusing the weights from diffusion models pre-trained on large-scall scale image datasets.
We work with 256 x 256 images. The VQ encoder reduces the spatial dimension to 64 x 64, the resolution at which the denoising model is trained. The commonly used input for single-frame inpainting with a U-Net diffusion model to simply concatenate the masked image, mask and noisy target image along the channel dimension [63]. This gives an input shape of (2d + 1, 64, 64) where d is the dimension of the latent VQ codes. This U-Net model stacks blocks of convolutions, followed by self-attention within the spatial features. VIDM also concatenates the noisy target zu with conditioning e (i.e. the context images, Ii-h...... It. and masks, ma 1 ... mzgeni) however, we stack context frames (i.e. / - h through - 1) along a new dimension. As such, our U-Net accepts a (2d + 1, h + 1, 64, 64) tensor, where h is the number of additional context frames. We perform convolutions on the h 1 sets of spatial features in parallel but allow attention across all frames at attention blocks. Consequently, we can re-use the weights of an inpainting diffusion model trained on
4

---

Table 1: In-painting evaluation on held-out clips from Epic-Kitchent [12]. Use of strong generative priors and past frames allows our model to outperform past works that use only one or the other.
Inpainting Method
PSNR
SSIM
FID
Runtime
Latent Diffusion [63]
28.29
0.931
27.29
12.5s / image
Latent Diffusion (fine-tuned)
28.27
0.931
27.50
12.5s / image
DLFormer [62]
26.98
0.922
51.74
106,4s / image
VIDM (Ours)
32.26
0.956
10.37
13.6s / image
VIDM trained with hands visible
31.81
0.953
11.04
13.6s / image
single images. The final spatial features are combined using (h+ + 1) x 1 convolution to form the final spatial prediction. Following prior work, the loss is (gg.c,e~V(0.1).e. [[le =0(=., U, c)]l]]-
Training Dataset. The data for training the model extracted from Epic-Kitchens [12] and a subset of Ego4D [20] (kitchen videos). Crucially, we don't 't have real ground truth (i.e. videos with and without human hands) for training this model. We synthetically generate training data by masking out hand-shaped regions from videos and asking the model to inpaint these regions. We use hand segment sequences from VISOR [13] as the pool of hand-shaped masks. In addition, we also generate synthetic masks using the scheme from [63, 104]. We use a 3-frame history (i.e. h = 3). Context images are drawn to be approximately 0. apart. We found it important to not have actual human hands prediction targets for the diffusion model. Thus, we omit frames containing hands from Ego4D (using hand detector from [70]) and do not back-propagate loss on patches that overlap a hand in Epic-K BitchKss (we use ground truth hand annotations on Epic-Kitchens frames from VISOR). We end up with 1.5 million training frames in total.
Model Training. We initialize our networks using pre-trained models. Specifically, we use the pre- trained VQ encoder-decoder from [63] which is kept fixed. The latent diffusion model is pre-trained for single-frame inpainting the Places [105] dataset and is finetuned for our multi- fratm--frmmeee inpainting task. We train with a batch size of 48 for 600k iterations on 8 A40 GPUs for 12 days. At inference time we use 200 denoising steps to generate images.
Our overall model realizes the two desiderata in the design of a video inpainting model. First, conditioning on previous frames allows occluded regions to be filled using the appearance from earlier frames directly (if available). Second, the use of a strong data-driven generative prior (by virtue of starting from diffusion model pre-trained on a large dataset) allows speculation of the content of the masked regions from the surrounding context.
Experiments
We design experiments to test the inpainting abilities of VIDM (Sec. 5.1), and the utility of our factorized representation for different visual perception and robot learning tasks (Sec. 5.2 to Sec. 5.6). For the former, we assess the contribution of using a rich generative prior and conditioning on past frames. For the latter, we explore benchmark tasks: object detection, 3D object reconstruction, affordance prediction, learning reward functions, and learning policies using learned rewards. We compare alternate representations, specifically ones used in past papers for the respective tasks.
5.1 Reconstruction Quality Evaluation
We start by assessing the quality of our proposed inpainting model, VIDM. Following recent litera- ture /16,41, 62], we evaluate the PSNR, SSIM and FID scores of inpainted frames against ground truth images.
Evaluation Dataset. We select 33 video clips that do not contain any hands from the held-out participants from the Sepic-Kicheee dataset [12]. These clips are on average 3.5 seconds long and span a total of 5811 frames. For each clip, we mask out a sequence of hand-shaped masks mined from the VISOR dataset [13]. The prediction problem for inpainting models is to reproduce the regions underneath the hand-shaped masks. This procedure simulates occlusion patterns that models will need to inpaint test time, while also providing access to ground truth pixels for evaluation.
5

---

a) Original Image
b) LafentDiffusion FT [63]
c) DLFormer [62]
d) VIDM (Ours)
Figure Our approach (VIDM) is able to correctly steal background information from past frames
(top row, oranges the bottom right) and also correctly reconstructs the wok handle using strong object appearance priors (bottom row).
Table 2: Average recall of detections from a COCO-trained Mask RCNN [24] on active objects (i.e. objects undergoing interaction) from VISOR [13]. See Sec. 5.2 for more details.
Image Used
AR @
AR @5 AR @
ARas @1 AR ARa5 10
Raw Image (l.e. I2)
0.137
0.263
0.272
0.265
0.530
0.551
Masked Image (i.e. J, with mi mi blacked out)
0.131
0.236
0.245
0.266
0.474
0.495
nes inpainted using Latent Diffusion [63] 4TM inpainted using Latent Diffusion (finetuned)
0.149
0,259
0.270
0.299
0.517
0.541
0.154
0,262
0.271
0.305
0.519
0.540
1T inpainted using VIDM (Ours w/o factorization)
0.163
0.268
0.277
0.317
0.521
0.543
and 15th inpainted using VIDM (Ours factorization)
0.170
0.379
0.411
0.334
0.681
0.735
Baselines. We compare against 3 baseline models: a) the single-frame latent diffusion inpainting model [63] trained on the Places [105] dataset that we initialize from, b) this model but finetuned on single images from the same KitihKis dataset that use for training, c) DLFormer [62], the current state-of-the-
video inpainting model. DLFormer trains one model per clip and sees every frame in the test clip, unlike model which only sees the previous 3 frames for each prediction. Finally, we include one ablation: VIDM trained with hands visible in the training objective (i.e. gradients propagate all pixels) instead of the objective used in VIDM (stop-gradient on pixels with hands, see Section 3 for details).
Results. Tab. reports the results. We can see our model outperforms all baselines. Processing context frames with attention allows our model to improve upon single-frame diffusion baselines. Training with multiple frames and hands visible improves performance, but this training procedure requires the model to output hands during training. Consequently, inpainted regions sometimes include hands at test time (see Figure S5). Our model also outperforms DLFormer [62], while being 8x faster and using only frames history. Fig. 4 shows example visualizations of inpainting results.
5.2 Application 1: Object Detection
The detection and recognition of objects being interacted with (by humans robots) is made challenging due to occlusion caused by the hand / end-effector. While existing object detection datasets may include objects held and occluded by human hands, objects may be occluded in ways outside of the training distribution (e.g. when running COCO-traines detectors on egocentric videos). Furthermore, very few object detection datasets include examples of occlusion by robot end-effector. We hypothesize that by removing the agent from the scene, off- the-shelf detectors may work better without requiring any additional training.
Protocol. We evaluate this hypothesis by testing COCO [39]-trained Mask RCNN detectors [24] on egocentric frames showing hand-object interaction from VISOR [13]. We only evaluate object

---

Table Average Precision on the gegiion-of-Ineeratinn task and mAP for Grasps Afforded by Objects task from [19]. Using our model to remove the hands improves performance vs. placing square mask over hands as done in [19]. See Sec. 5.4 for more details.
Region of Interaction (Rol) 0% Slack
Grasps Afforded by Objects (GAO)
Image Used
1% Slack
Masked image (1, with m blacked out) [19] Ie (inpainted using VIDM) (Ours)
47.50 + 2.77
54,03 1 3.27
35.53 + 3.67
50.77 = 0.81
57.10 + 1.01
41.00 + 3,99
classes that appear in both COCO and VISOR. VISOR [13] only annotates the active object and background objects have not been annotated. Thus, we measure performance using Average Recall (AR) as opposed to Average Precision. Specifically, we predict & (= 1, 5, 10 in our experiments) boxes per class per image and report box-level Average Recall at 0.5 box overlap (ARos) and integrated over box overlap thresholds {0.5, 0.55, .... 0.95} (ARai).
Results. Tab. 2 reports the average recall when using different images as input to the SOCO-traaine Mask RCNN detector. We compare against a) just using the raw image (i.e. I,) as would typically be done, b) using masked image (i.e. I, with m blacked out) naive way to remove the hand, c) different methods for inpainting the agent pixels (pre-trained and finetuned single-frame Latent Diffusion Model [63] and our inpainting scheme, VIDM), and d) using both the original image and the inpainted image (i.e. both 1, and T(=))) as enabled by AEF. When using both images, we run the pre-trained detector on each image, merge the predictions, and return the top-k.
Naive masking introduces artifacts and hurts performance. Inpainting using a single-frame frame model helps but not consistently. Our video-based inpainter leads to more consistent gains over the raw image baseline, outperforming other inpainting methods. Our full formulation leads to the strongest performance achieving 24%-51% relative improvement over the raw image baseline.
5.3 Application 2: 3D Reconstruction of Hand-Held Objects
Past work has tackled the problem of 3D reconstruction of hand-held objects [22, 34,59,99, Here again, the human hand creates occlusion and hence nuisance, yet it provides valuable cues for the object shape [95]. Similar to Sec. 5.2, we hypothesize that the complete appearance of the object, behind the hand, may provide more signal to 3D reconstruction model.
Protocol. We adopt the state-of- -ote-e--rttt---ttt approach from Ye et al. [95]. They design custom neural network architecture that is trained with full supervision on the ObMan dataset [22]. Their model just accepts 4, as input. To test whether the complete appearance of the object is helpful, we additionally input 1HTh to their model. As ObMan is synthetic dataset, we use ground-truth 11 images obtained by rendering the object without the hand. This evaluation thus only measures the impact of the proposed factorization. We use the standard metrics: F1 score at 5 & 10 mm, and chamfer distance.
Results. Using I, and ground truth IeTh improves upon just using the raw image (as used in [95]) across all metrics: the Fl score at 5mm increases from 0.41 to 0.48, the Fl score at 10mm increases from 0.62 to 0.68, and the chamfer distance improves from 1.23 to 1.06. As we are using ground truth Tew here, this is not surprising. But, it does show the effectiveness of our proposal of using factorized agent-enviromment representations.
5.4 Application 3: Affordance Prediction
Past works [19] have used videos to learn models for labeling images with regions of interaction and afforded grasps. While egocentric videos directly show both these aspects, learning is made challenging because a) the image already contains the answer (the location and grasp type exhibited by the hand), and b) the image patch for which the prediction needs to be made is occluded by the hand. Goyal et al. [19] mask out the hand and train models to predict the hand pixels from the surrounding context. This addresses the first problem but makes the second problem worse. After masking there is even more occlusion. This application can be tackled using our AEF framework. where we use inpainted images 1aw to predict aspects of the agent shown in 1 "gge
7

---

Table Spearman's 's rank correlations of reward functions learned on Epic- Kitchens [12] when evaluated on robotic gripper sequences for opening drawer, cupboard, and fridge tasks. Our factored representation achieves better performance than raw images or environment-oniy representation.
Input Representation
Drawer
Cupboard
Fridge
Overall
Raw Imges (i.e. 1)
0.507
0.507
0.660
0.558
Inpainted only (i.e. Iew as proposed in [1])
0.574
0.570
0.671
0.605
AEF (Tem and (T) (Ours)
0.585
0.582
0.676
0.614
Protocol. We adopt the training scheme from [19] (current state-of-the-art on this benchmark), but instead of using masked images as input to the model, we use images inpainted using our VIDM model. We adopt the same metrics and testing data used in [19] and report performance on both the Region of Interaction (Rol) and Grasps Afforded by Objects (GAO) tasks. We test in a small data regime and only use 1000 images for training the affordance prediction models from [19]. We report mean and standard deviation across 3 trainings.
Results. Tab. 3 presents the metrics. Across both tasks, the use of our inpainted images leads to improved performance vs. using masked-out images.
5.5 Application 4: Learning Reward Functions from Videos
Difficulty in manually specifying reward functions for policy learning in the real world has motivated previous work to learn reward functions from human video demonstrations [1, 6, 9, 71,. The visual mismatch between the human hand and robot end-effector is issue. Past work [1] employs inpainting to circumvent this but consequently loses the important signal that the hand motion provides. Our factored representation in AEF provides more complete solution and, as we will show, leads to improved performance over just using the raw data [9, 71] and inpainting out the hand [1].
Protocol. We train a reward predictor on video clips from of the reward predictor on trajectories from a robotic end-effector. Specifically, we consider three tasks: opening drawer, opening a cabinet, and opening a refrigerator. We use the action annotations from [12] to find video segments showing these tasks to obtain 348, 419, and 261 clips respectively. The reward function is trained to predict the task progress (0 at clip start and at clip end) from single image. The learned function is used to rank frames from trajectories of robotic end-effector, specifically the reacher-grabber assistive tool from [76, 98] captured using a hean-mounted GoPro HER07. We collected 10, 10, and trajectories respectively for drawers, cupboards, and refrigerators. These robotic trajectories represent large visual domain gap from the human-handed training demonstrations and test how well the learned reward functions generalize. Following [7], we measure Spearman's 's rank correlation between the rewards predicted by the learned reward function and the ground truth reward (using frame ordering) on the pseudo-robotic demonstrations. The Spearman's 's rank correlation only measures the relative ordering and ignores the absolute magnitude.
[12] and assess the quality
Results. Tab. 4 reports the Spearman's 's rank correlations for the different tasks. We compare the use of different representations for learning these reward functions: a) raw images (i.e. I,), b) just inpainting (i.e. Te and c) our factored representation (i.e. I.em and rageu)) ). For our factored representation, we map pagent through a function to an agent-agnostic representation by simply placing green dot the highest point on the end-effector mask (human hand or robot gripper) in the image plane (see Fig. 5), thus retaining information about where the hand is in relation to the scene. Our factored model, ABF, correlates the best with ground truth. Just painting out the end-effector loses important information about where the end-effector is in relation to objects in the scene. Raw frames maintain this information but suffer from the visual domain gap.
5.6 Application 5: Real-Worle Policy Learning using Learned Rewards
Motivated by the success of the offline evaluations in Sec. 5.5, we use the learned reward function for opening drawers to learn drawer opening policy the real world using a Stretch RE2 robot (see Fig. 5 (left)).
8

---

1.0
Uy and g(( (Ours)
Imges
0.8
Inpainted only (l.. ff as propesed in [1]
Surcess Rate
0.2
0.0
CEM lterations
Figure 5: Real-world experiment setup and results. (left) Raw views from camera, (center) Inpainted image with agent-agnostic representation (green dot top-pixel of end-effector). (right) Success rate as function of CEM iterations.
Protocol. We position the robot base at a fixed distance from the target drawer. We use a 1D action space that specifies how far from the base the hand should extend to execute grasp, before retracting. The drawer opens the robot able to correctly grasp the handle. We use GoPro camera to capture RGB images for reward computation. The total reward for each trial is the sum of the predicted reward at the point of attempted grasp, and after the arm has been retracted. We use the Cross- cooss-Entropy Method (CEM) [14] train the policy. We collect 20 trials in each CEM iteration and use the top-7 as elite samples. Actions for the next CEM iteration are sampled from a Gaussian fit over these elite samples. We use the success rate at the different CEM iterations as the metrie.
Results. Similar to Sec. 5.5, we compare against a) raw images (i.e. I,), b) inpainted images (i.e. just 1eTM as per proposal in [1]), and c) our factored representation, AEF, As before, we use dot at the end-effector location as the agent-agnostic representation of ragent We use our VIDM for inpainting images for all methods. We finetune DeepLabV3 [9] semantic segmentation model (on 100 manually annotated images) to segment out the Stretch RE2 end-effector. Fig. (right) shows the success rate as function CEM iterations. Reward functions learned using our AEF representation outperform baselines that just the raw image or don't 't use the factorization.
6
Discussion
Our experiments have demonstrated the effectiveness of our inpainting model. Specifically, the use of strong priors from large-scale pre-trained inpainting diffusion models and the ability to steal content from previous frames allows VIDM to outperform past methods that only use one or the other cue (pre-trained LDM [63] and DLFormer [62] respectively). This is reflected in qualitative metrics in Tab. and also in qualitative visualizations in Fig. 4.
Our powerful video inpainting model (along with semantic segmentation models that can segment out the agent) enables the extraction of our proposed Agent-Environnin Factored (AEF) representation from egocentric video frames in pixel space. This allows intuitive and independent manipulation of the agent and environment representations, enabling a number of downstream visual perception and robotics applications. Specifically, we demonstrated results on 5 benchmarks spanning categories. For visual perception tasks (object detection and 3D reconstruction of hand-held objects), we found additionally using the environment representation, which has the agent removed, improved performance. For affordance prediction tasks (region of interaction and grasps afforded by objects), we found that using the inpainted environment representation to be more effective at predicting relevant aspects the agent representation than naive masking of the agent as used in past work. For robotic tasks (learning rewards from videos), the factored representation allowed easy conversion of the agent representation into an agent-agnostic form. This led to better transfer of reward functions across embodiments than practices adopted in past work (using the original agent representation and ignoring the agent altogether) both in offline evaluations and online learning on a physical platform.
Given that all information comes from the masked input frames, one might wonder, what addi- tional value does the proposed inpainting add? First, provides practical advantage that each downstream application doesn't 't need to separately pay the implicit training cost for interpreting

---

masked-out images. Inpainted images are like real images, allowing ready re-use of the off-the-shelf models. Furthermore, the data complexity of learning a concept under heavy occlusion may be much higher than without. A foundation inpainting model can leverage pre-training on large-scale datasets, to inpaint the occlusion. This may be more data-efficient way to learn concepts as our experiments have also shown.
Limitations and Broader Impact
As with most work with diffusion models, inference time for our method is 13.6 seconds which renders real-time use in robotic applications infeasible. AEF only inpaints out the hand to recover the unoccluded environment. There may be situations where the environment might occlude the agent and it may be useful to explore if similar inpainting could be done for agent occlusion.
At broader level, while our use of the video inpainting model was for robotic applications, the VIDM model can also be thought of a general-purpose video inpainting model. This inherits the uncertain societal implications of generative Al models.
Acknowledgments and Disclosure of Funding
We thank Ruisen Tu and Advait Patel for their help with collecting segmentation annotations for the claw and the robot end-effector. We also thank Arjum Gupta, Erin Zhang. Shaowei Liu. and Joseph Lubars for their feedback on the manuscript. This material is based upon work supported by NSF (ISS000735), DARPA (Machine Common Sense program), NASA (80NSSC2IK1030) an Amazon Research Award, an NVidia Academic Hardware Grant, and the NCSA Delta System (supported by NSF OCI 2005572 and the State of Illinois).
References
[1] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. In RSS, 2022. 1, 8, 8,9,1 17
[2] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as versatile representation for robotics, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1,2
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv:2211.011 324, 2022. 3
[4] Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro, and Stanley Osher. Simultaneous structure and texture image inpainting. In Proceedings of the IEEE Conference Computer Vision and Pattern
Recognition (CVPR), 2003. 3
[5] Angel Chang. Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li. Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet:
An information-rich 3d model repository. arXiv, 15 151003301, 2015. 16
[6] Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos. In NewrlPS, 2020.
[7] Matthew Chang, Arjun Gupta, and Saurabh Gupta. Learning value functions from undirected state-only experience. In International Conference on Learning Representations, 2022. 8, 17
[8] Annic S Chen, Suraj Nair, and Chelsca Finn. Learning generalizable robotic reward functions from" the-wwid" human videos. arXiv preprint arXiv:2103. 16811, 2021.
[9] Liang-Chion Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolu- tion for semantic image segmentation. arXiv preprint arXiv: /tt.:::/700.5/88/// 2017. 4, 8,9
[10] Enric Corona, Albert Pumarola, Guillem Alenyà, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-ohject scenes. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR). 2020. 16
[11] Antonio Criminisi, Patrick Perez, and Kentaro Toyama. Object removal by exemplar- based inpainting. In Proceedings the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2003. 3
10

---

[12] Dima Damen, Hazel Doughty. Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos. Davide Moltisanti, Jonathan Munro, Toby Perrett Will Price, and Michael Wray. Rescaling egocentric vision:
Collection, pipeline and challenges for epic-kitchens- 100. International Journal of Computer Vision (NCV), 130:33-55 55, 2022. 5, 8, 16
[13] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. In Proceedingy of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks,
2022. 2, 2,4.3 5, 6, 7, 15, 16
[14] Pieter-Tjerk De Boer, Dirk Kroese, Shie Mannor, and Reuven Rubinstein. A tutorial on the cross- entropy method. Annals of operations research, 134:19-67, 2005. 9
[15] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances
Neural Information Processing Systems (NeurIPS), 2021.
[16] Tim Dockhom, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically langevin diffusion. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[17] Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In Proceedings the seventh IEEE international conference on computer vision, volume 2, pages 1033-1038. IEEE, 1999.
3
[18] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high resoluesooution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021. 4
[19] Mohit Goyal, Sahil Modi, Rishabh Goyal, and Saurabh Gupta. Human hands as probes for interactive
object understanding. In Computer Vision and Pattern Recognition (CVPR), 2022. 1, 2, 3, 7, 8, 16
[20] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray. Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer. Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang. Yifei Huang. Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari. Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi. Yunyi Zhu, Pablo Arhelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park. James M. Rebg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan. and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022. 5, 15
[21] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. In Advances Neural Information Processing Systems (NeurIPS).
2022. 3
[22] Yana Hasson, Gul Varol, Dimitrios Tzionas, Tgor Kalevatykh, Michael Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction hands and manipulated ohjects. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 1 1807-11816, 2019. 7, 16, 17
[23] James Hays and Alexei A Efros. Scene completion using millions of photographs. ACM Transactions on Graphics (ToG), 2007. 3
[24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. 6, 16
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), arXiv preprini 168:0.10000004
2016.
[27] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv:2208.01626 2022. 3
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 13:6650-8551, 2020. 3,4

---

[29] Jonathan Ho. Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi. and Tim Salimans. Cascaded diffussin models for high fidelity image generation. Journal of Machine Learning Research, 23:47:1-47:333 2022. 3
[30] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.
Video diffusion models. arXiv:2204.03458. 2022. 3
[31] Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. arXiv preprint XXix:2206.076990 2022. 3
[32] Ra-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf. Temporally coherent completion of dynamic vidco. ACM Transactions on Graphics (TOG), 35(6):1-11, 2016.
[33] Satoshi lizuka, Edgar Simo-Serra and Hiroshi Ishikawa. Globally and locally consistent image completion.
ACM Transactions on Graphics (ToG), 36(4):1-14, 2017.
[34] Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In 2020 International Conference on 3D Vision (3DV), pages 334. IEEE, 2020.7 7
[35] Dahun Kim, Sanghyun Woo, Joon- Young Lee, and In So Kweon. Deep video inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3
[36] Sungho Lee, Scoung Wug Oh, Dae Yeun Won, and Scon Joo Kim. Copy-and-paste networks for deep video inpainting. In Proceedingy of the IEEF/CVF International Conference on Computer Vision, pages
1413-44214 2019. 3,5
[37] Ang Li, Shanshan Zhao, Xingjun Ma, Mingming Gong, Jianzhong Qi, Rui Zhang, Dacheng Tao, and Ramamohanarao Kotagiri. Short-term and long-term context aggregation network for video inpainting. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 3
[38] Jingyuan Li, Ning Wang. Lefei Zhang, Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR).
2020. 3
[39] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pages 740-755. Springer, 2014. 4, 6, 16
[40] Guilin Liu, Fitsum A Reda, Kevin Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[41] Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fuseformer: Fusing fine-graines information transformers for video inpainting, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14040-14049, 2021.
3.5
[42) Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3282-3292, 2022. 2
[43] Andreas Lugmayr. Martin Danelljan, Andres Romero, Fisher Yu. Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461-11471, 2022. 3
[44] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed, arXiv, 2101,02388, 2021.
[45] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3
[46] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS). 2016.7
[47] Yuqing Ma, Xianglong Liu, Shihao Bai, Lei Wang, Aishan Liu, Dacheng Tao, and Edwin R Hancock. Regionwise generative adversarial image inpainting for large missing areas. IEFE Transactions on Cybernetics, 2022. 3
[48] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In Proceedings of the International Conference Learning Representations (ICLR), 2021. 3
[49] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- 106, 2021. 17
12

---

[50] Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8688-8697, 2019.
[51] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal
visual representation for robot manipulation. arXiv preprint arXiv:2203.1266)) 2022. 3
[52] Kamyar Nazeri, Eric Ng. Tony Joseph, Faisal Z. Qureshi, and Mchran Ebrahimi. Edgeconnect: Generative
image inpainting with adversarial edge learning. arXiv preprint arXiv: /0ti::/90/0000/// 2019.
[53] Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. Video inpainting complex scenes. Siam journal on imaging sciences, 7(4): 1993-299,,,,, 2014.
[54] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. Proceedings of the International Conference on Machine Learning (ICML), 2021. 3
[55] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the International Conference on Machine Learning (ICML), 2021. 3
[56] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE
Conference Computer Vision and Pattern Recognition (CVPR), 2019. 17
[57] Austin Patel, Andrew Wang, Iija Radosavovic, and Jitendra Malik. Learning to imitate object interactions from internet videos. arXiv preprint 2t 2022. 1
[58] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536-2544, 2016. 3
[59] Aditya Prakash, Matthew Chang, Matthew Jin, and Saurabh Gupta. Learning hand-held object recon- struction from in-the-wild videos. arXiv, 2305.03036, 2023. 2, 3,7
[60] Iljaa Radosavovic, Tete Xiao, Stephen James, Picter Abbeel, Jitendra Malik, and Trevor Darrell. Real- world robot learning with masked visual pre-training. In Conference on Robot Learning, pages 416-426, PMLR, 2023. 3
[61] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv:2204.00 0225, 2022. 3
[62] Jingjing Ren, Qingqing Zheng. Yuanyuan Zhao, Xuemiao Xu, and Chen Li. DIformer: Discrete latent transformer for video inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022. 2, 3, 6, 6, 20
[63] Robin Rombach. Andreas Blattmann. Dominik Lorenz. Patrick Esser, and Björn Ommer. High-resolution image syathesis with latent diffusson models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022. 3, 4. 5, 6, 7, 15, 20
[64] Javier Romero, Dimitrios Tzionas, and Michael Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics (ToG), 2017.
[65] Yu Rong, Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3D hand and body motion capture by regression and integration. Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops), 2021. 17
[66] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Asse Assisted Intervention-MICCAl 2015: I 8thh International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Sprinee, 2015. 3,4
[67] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022. 3
[68] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. 3
[69] Karl Schmeckpeper, Olch Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsca Finn. Reinforcement lcarning with videos: Combining offline observations with interaction. arviv preprint arXiv:2011 2020. 3
[70] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Foubey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2.5
13

---

[71] Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2robot: Learning, manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Rexearch, 10(12-14):1419-4433
2021.
[72] Yuan Shen, Wei-Chiu Ma, and Shenlong Wang. SGAM: building a virtual 3d world through simultancous generation and mapping. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3
[73] Laura Smith, Nikita Dhawan, Marvin Zhang, Pieter Abbeel, and Sergey Levine. Avid: Learning multi- stage tasks via pixel-level translation of human videos. arXiv preprint 1rXiv:1912.04443, 201094444,,,,,....... 3
[74] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference on Learning Representations (ICLR), 2015. 3
[75] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. 3
[76] Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser. Crasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations, IEEE Robotics and Automation Letters, 5(3):4978- 4985, 2020. 8
[77] Yang Song, Jascha Sohl-Dicks -iickteein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[78] Yuhang Song, Chao Yang. Yeji Shen, Peng Wang. Qin Huang, and C-C Jay Kuo. Spg-net: Segmentation
prediction and guidance network for image inpainting, a 03356, 2018. 3
[79] Michael Strobel, Julia Diebold, and Daniel Cremers. Flow and color inpainting for video completion. In German Conference on Puttern Recognition, pages 293-304. Springer, 2014. 3
[80] Roman Suvorov, Elizavcta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Alcksei Silvestrov, Naejin Kong. Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robut large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applicaations of computer vixion, pages 2149-2159, 2022.
[81] Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3D distillation of self supervieee 2D image representations. In Proceedings of the International Conference on 3D Vision (3DV), 2022. 3
[82] Vadim Tschernezki, Diane Larlus, and Andrea Vodaldi. Neuraldiff: Segmenting 3d objects that move in egocentric videos. In 202/ International Conference 3D Vision (3DV), pages 910-919, IEEE, 2021. 3, 17
[83] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors. Advances in Neural Information Processing Systems (NeurlPS), 2021. 3
[84] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neurul information processing systems, 30, 2017. 4
[85] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 4
[86] Vikram Voleti, Alexia Martineuu and Christopher Pal. Masked conditional video diffusion for prediction, generation, and interpolation. arXiv:2205. 00853, 2022.
[87] Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. Video inpainting by jointly learning temporal structure and spatial details. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[88] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In Proceedings of the International Conference on Learning Reprexentations (ICLR), 2022. 3
[89] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https:
//g thub. com//facebookrsseart ch/ detectron2, 2019. 16
[90] Zhiliang Wu, Hanyu Xuan, Changehang Sun, Kang Zhang, and Yan Yan. Semi-supervised video inpainting with cycle consistency constraints. arXiv preprint arXiv:2208.0680) 2022. 3
[91] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre- pra-trainin for motor control. arXiv preprint 203iv:2203.06173, 2022.
[92] Haoyu Xiong, Quanzhou Li, Kun-Chun Chen, Homanga Bharadhwaj. Samarth Sinha, and Animesh Garg.
Learning by watching: Physical imitation of manipulation skills from human videos. In 2021 IEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 7827-7834, IEEE, 2021.
14

---

[93] Wei Xiong, Jiahui Yo, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo. Foreground-awand image inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3
[94] Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Deep flow-guided video inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3
[95] Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022. 2, 3, 7, 16, 17
[96] Yufei Ye. Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song. Shubham Tulsiani, and Sifei Liu. Affordance diffuusio: Synthesizing hand-object interactions. In CVPR, 2023. 3
[97] Zili Yi, Qiang Tang, Shekoofch Azizi, Daesik Jang, and Zhan Xu. Contextual residual aggregation for ultra regoureeoottion image inpainting. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020. 3
[98] Sarah Young. Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual
imitation made easy. 2020. 8
199] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4471-4480, 2019. 3
[100] Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, pages 537-546. PMLR, 2022.
[101] Xiaohui Zeng, Arash Vabdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis.
LION: latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems (NeurlPS), 2022. 3
[102] Yanhong Zeng. Jianlong Fu, and Hongyang Chao. Learning joint spatial-temporal transformations for
video inpainting. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 3
[103] Lingzhi Zhang. Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXIX, pages 127-145. Springer, 2022. 2,3
[104] Shengyu Zhao, Jonathan Cui. Yilun Sheng. Yue Dong, Xiao Liang. Eric Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2103.1442 2021. 3,5
[105] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1445 2017. 5,6
[106] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point- voxel diffusion. Proceedings of the IEFF International Conference on Computer Vision (ICCV), 2021. 3
[107] Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang. Learning to generate realistic lidar point clouds. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors. Proceedings of the European Conference on Computer Vision (ECCV), 2022. 3
S1 VIDM Training Details
For an overview of the VIDM model architecture, see Section 4. In each block, the CNN layers are implemented as residual blocks [25] with SiLU non-linearities [26] and each attention layer does self-attention across all token from all input images using 32-channel GroupNorm normalization. Following [63], upsampling and downsampling operations are both implemented using residual CNN blocks with either an internal nearest mode 2x upsampling operation or internal 2 downsampling via average pooling. An initial convolution brings the feature dimension to 256, which is raised to a maximum of 1024 at the center of the U-Net. At the highest spatial resolution of 64 x 64 the self-attention layer is omitted, as attention with 16384 (= 64 x 64 x 4) tokens is computationally intractable for our available hardware. The largest attention layer occurs at a spatial resolution of 32 32 across four images for total of 4096 tokens.
We trained VIDM using target images from Ego4D [20] and VISOR [13] (see Section 4). Since no evaluation was done on Ego4D, no Ego4D data was held out. For VISOR, all data from participants
15

---

P37, P35, P29, POS, and P07 was held-out from training. This held-out data from these participants was used for reconstruction quality evaluation (Section 5.1) and object detection (Section 5.2)
experiments. Table S1 lists hyper-parameters. Figure S3 shows sample training batches.
Table S1: VIDM Model and Training
Hyper-parameter
Value
Learning Rate
4.8 x 10
Batch Size
48
Optimizer
Adam
Diffusion Steps (training)
1000
Latent image Size
64 64
Number of VQ Emhedding Tokens
8192
VQ Embedding Dimension Diffusion Steps (inference)
3
200
Attention Heads
8
S2 Downstream Task Experimental Details
$2.1
Detection
We used off-the-shelf Mask R-CNN R_ 101 S____P_333_ __ from Detectron2 [24, 89] trained the COCO dataset [39] for evaluation. We used overlapping classes between the VISOR [13] annotations and COCO for evaluation. These were: apple, banana, bottle, bowl, broccoli, cake, carrot, chair, cup, fork, knife, microwave, oven, pizza, refrigerator, sandwich, scissors, sink, spoon, toaster.
S2.2 Affordance Prediction
We experiment on EPIC-ROI and GAO tasks from Goyal et [19]. EPIC-ROI uses the EPIC-KITCHENS dataset [12] and GAO uses (CB-Affordance [10] dataset. We consider a low data regime in our work and sample 1K images from these datasets to train the different models. For EPIC-ROI, we sample images with probability inversely proportional to the length of the video. For GAO, we sample randomly. use the same evaluation setting from [19].
Dataset:
We use the same architecture from ACP [19] and replace the EPIC-ROI input images with images produced by our inpainting model (with hands removed) incorporate
Model:
factorized representation. While ACP [19] masks out a patch at the bottom center of the image to hide the hand, do not need any mask (neither for training for testing) since the hands have been removed via inpainting. The input is processed by ResNet-50 followed by different decoders for EPIC-ROI and GAO tasks.
Training: We train separate models for EPIC-ROI and GAO using the loss function and hyperparam- eters from ACP [19]. While is possible to train single model in multitask manner, observe that the two tasks are not complementary to each other. We train using 3 seeds for each task and report the mean and standard deviation in the metrics.
S2.3
3D Reconstruction of Hand-held Objects
We use ObMan [22] dataset which consists of 2.5K synthetic objects from ShapeNet [5]. use train and test splits provided by Ye et al. [95]. We divide the train split into train and set. The train set consists of 134K, val set 7K and test set 6.2K images. The dataset provides 3D CAD models for each object, which we use for training hand-held object reconstruction model from Ye et al. [95].
Dataset:
16

---

Input
NeuralDiff
VIDM
Inpat
NeuralDiff
VIDM
Figure S1: The environment factorization obtained by NeuralDiff contains artifacts, e.g. (left) region around banana, (right) partial hand. Our VIDM model produces more realistic results.
Model: We use the architecture from Ye al. [95]. It uses FrankMocap [65] extract hand articulation features from a single image using MANO [64] hand parameterization. These hand features are used as conditioning to DeepSDF [56] model which predicts the object shape using implicit representation. This model also takes in pixel-aligned features and global image features along with hand features. To incorporate our factorized representation, we also extract global image features and pixel-aligned features from ObMan images showing only objects (with hands removed). These features are concatenated with the features from the input ObMan images and fed as input to the DeepSDF [56] decoder.
Training:
Following [95], use normalized hand coordinate frame for sampling points and predicting SDFs. We sample 8192 points in [-1, 1] for training, out of which half of them lie inside and the rest lie outside the object. At test time, 64 points are sampled uniformly in [-1, 1]3. We train the model in supervised manner using 3D ground truth from ObMan [22] for 200 epochs with a learning rate of 1e- Other typer-parameters are used directly from [95].
S3
Additional Results
S3.1 NERF Comparison
Neural radiance fields [49] represent an alternative path to producing agent-environment factorizations. We compare against NeuralDiff [82] as a representative method. We can only compare on the EPIC-Kitchens PO5_0 01 sequence since it is the only one model released from that project that is common with our test set. We focus on frames that include a hand, and use their static and transient reconstruction as the prediction for /eav We contrast it with the prediction for from model.
Figure $1 shows qualitative comparisons. On these images, our model achieves superior FID scores 186.79 for VIDM vs 215.90 for NeuralDiff. Note that FIDs are overall higher than usual, but for good reason. There no hand-removed image set (ic. objects floating in air) to use as reference to compute FID. As proxy reference set, we use images that don't 't contain hands as reference and thus FID scores for both models are higher than usual.
S3.2 Robot Learning: Pick-Up-Plate
To demonstrate that our method can work on tasks beyond opening, we followed the same protocol as in Table 4 for a fourth task of picking up plate. In epic kitchens, there are less than I 3 as many sequences for this task as for opening drawers, and the quality worse (clips having the plate out of frame, annotation timing being off etc.). This lack of data and low quality hurts generalization for all methods, but we still see positive trend where using VIDM inpainted images with factorization gives Spearman's 's correlation of 0. 0319 while raw images and non-factorized inpainting give 0.118 and 0.083 respectively. We note that many other cross embodiment learning techniques may be used with our factored representation to explore more complex multi-stage tasks (e.g [1, 7]) which leave to future work.
S3.3 Average Precision for Object Detection
In addition to the recall results reported in Table 2 we performed an small experiment to access methods effect on object detection precision. To this end took the class with the fewest
17

---

h-3 1 4-11
I
I in- in-painted (oulput)
When the oranges are occluded by the hand, but visible in the previous frame, VIDM is able to reconstruct the oranges using this information (first row). When the oranges in previous frames covered by either black red masks, the model fails reconstruct the oranges and inpaints red or black pixels instead (second and third rows). This suggests that the model does refer content from the context frames for in-painting.
Figure S2:
false positives (which happened to be 'scissors') when using raw images, and manually labeled all instances which were indeed true positives (adding missing detections to the ground truth labels). For this class with labels updated, using raw images only achieves AP of 0.738, while using images in painted with VIDM achieves an AP 0.762.
S4 Visualizations
In Figure S3, we include visualization of training batch for our method, showcasing supervision and generated masks. In Figure S4, we include additional visualizations of the predictions made by our method and baselines.
In Figure S2 we visualize how VIDM responds to corruptions in the input images to highlight VIDM's ability to copy information from the context images. For this visualization images where the region occluded by the hand visible in the context frames. We manually corrupt the context frames by masking (with both black and red boxes) the region from which the model needs to draw information to inpaint properly. We see that when presented with corrupted context frames, VIDM copies the corrupted pixels (black red) represent the occluded region, while properly inpaints the region when using the uncorrupted frames.
sequence of
Qualitative visualizations in Figure S5 exhibit the failure mode of training with hands visible. Because this training procedure requires the model to output hands some of the time, it sometimes paints the hand like pixels back into the image.
18

---

Column
Column 2
Column 3
Column 4
Column 5
Column Column
Figure S3: An example batch for training VIDM. Columns 1-4: Input images to the network. Column 5: target image for reconstruction. Column 6: Masked regions the target image. Column 7: Pixels with loss propagated (white pixels have loss, gray pixels have no loss). Note that hands that are masked in the target image (column 5) have no loss on them. See Section 4for details.
19

---

LF
a) Original Image
b) LatentDiffusion FT [63]
c) DLFormer [62]
d) VIDM (Ours)
Figure $4: Additional visualizations of predictions from our method and baselines.
20

---

Training /w hands
VIDM
Latent Diffusion
Training while propagating loss from pixels containing the hand, the model learns to reconstruct hands in occluded regions (left). This behavior does not occur with our training procedure, which does not apply loss on pixels with hands (center), and does not appear base single-image model that has no egocentric specific finetuning (right).
Figure S5:
21


Reviews:
[{"review_id": "P96ZNwfYJV", "reviewer": "Reviewer_UX1K", "summary": "The paper proposes agent-environment factorization (AEF) as a representation for egocentric videos. AEF consists of 2 parts: the hand segmentation as agent part, and video inpainted environment part. The former uses an off-the-shelf hand segmentation while the latter is from finetuning an inpainting diffusion model. The authors show several downstream applications to demonstrate the AEF can improve recognition, reconstruction, and robotic tasks. \n", "questions": "Overall I think it is a sound paper. The novelty are faire and mainly empirical -- showing this factorization can help several downstream tasks. See weakness. My main concern is 1.1 and 2. ", "limitations": "The author discussed the limitation explicitly. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The key idea is agent-environment factorization, which is shown beneficial to multiple applications, 2D recognition, 3D reconstruction, and robotic tasks. The baselines are carefully designed such that they are directly comparable and showcase where and why AEF helps.\n+ In the design of video inpainting model, they use the pretrained image inpainting to get the spatial prior while using nearby frames to aggregate more context. \n", "weaknesses": "1. While the factorization is shown effective, the improvement over the video inpainting model itself appears a bit incremental. This is based on 1) qualitative results in FigS3 2) some domain specific knowledge being used (e.g. no loss on the hand pixels, copy paste hand-shaped masks, etc). It is not clear how well the proposed architecture can generalize to other well-adopted video inpainting benchmark benchmarks like DAVIS or Youtube VOS. \n\n    1.1 The proposed architectures may not be as critical as the paper claims (like in Table 1, see 1.2). It seems finetuning an generative video model, e.g. a MAGVit or video diffusion model may lead to results as good as the current method shows. I wonder if the authors agree with my conjecture. \n\n    1.2 The numbers in Table 1 indicate a significant gap over current SoTA video inpainting methods. Improvement over Latent Diffusion is sensible since the model sees more context. But DLFormer is a completely different method – a per-clip model that operates in pixel space.  All of the metrics would favor sharp high-frequency signals, which latent space in latent diffusions are good at but DLFormer lacks. \n\n    1.3 visualizing attention may be helpful to understand how context in nearby frames help inpainting. \n\n2. The paper is very related to “Neural Feature Fusion Fields” which factorize the videos into background, agent, and, in addition, moving objects. Although they optimize a per-clip representation, the differences with this line of works should be discussed. \n\n3. There are some improvements across multiple downstreaming applications but the improvements are not surprising. For example, seeing the unoccluded objects / environment improves 3D reconstruction; merging prediction of both unoccluded and original images boosts object detection; seeing the hand location indicates frame order better. \n    3.1 It is a minor point but in Table 2, it will be more fair to compare if the proposed region is also doubled for the baselines since the proposed method is evaluated with twice the predictions.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and thoughtful feedback. Please see our response below and refer to the supporting figures in the rebuttal PDF.\n\n**Clarifications about improvement in the video inpainting model**\nFirst, to clarify, VIDM includes a) an architectural modification on top of an image-based diffusion model to use information from past frames via cross-frame attention, and b) domain specific insights to train VIDM for the application we are interested in (agent-environment factorization). In our view, improvements over the baselines are quite salient, both qualitatively and quantitatively. Our output in Figure S3 suffers from fewer artifacts (row 1, row 2, row 5, row 6) and better completes the objects (row 3, row 4, row 7). Quantitative improvements in Table 1 are also quite solid, PSNR improves from 28.27 to 32.26, FID decreases from 27.50 to 10.37. Note that baselines were retrained on the same dataset that we use, so this represents a solid improvement. \n\nIn the setting that we care about (egocentric videos), the major challenges are large camera motion, and dynamic occlusion (hands holding objects and moving through the scene). In this setting we are able to outperform the current state-of-the-art (DLFormer). Since our downstream applications involve hand occlusion, we focus on the task of inpainting hands in egocentric video. For this use case, our model outperforms existing models and lets us build effective agent-environment factorizations.\n\n**Discussion about finetuning a video generative model**\nThe reviewer’s suggested approach is feasible, but it is not trivial to apply in our setting. For starters, published video diffusion models at the time of submission  [19, 26, 75], could not be directly applied to our video inpainting task (they either operate at lower resolution or do not incorporate masks for inpainting). The reviewer’s suggested approach (taking a video prediction model, and adapting it to do inpainting), is similar to the approach we took (taking an inpainting model and adapting it to video). We started from an image-based generative model and made necessary modifications to get it to work for video in-painting. We had our reasons: LDM released code, code for MAGVit wasn’t available, other video generative models weren’t applicable for reasons above. So, yes we agree with your conjecture, and such alternatives are interesting avenues for future work. But that doesn’t nullify our contributions in this paper, that is AEF combined with VIDM.\n\n\n**Comparison with SoTA video inpainting methods.**\nIndependent of how DLFormer works, we compared against DLFormer as it is, to the best of our knowledge, the state-of-the-art at the video in-painting task. PSNR and SSIM are standard metrics. DLFormer itself uses these metrics for which it reports state-of-the-art results. We also report FID which is not directly comparing any two images, but rather comparing image statistics computed across all inpainted test clips versus ground truth. We outperform the current state-of-the-art for video inpainting in our setting (hand removal in egocentric videos) and our contribution (video inpainting extension of latent diffusion models) leads to improvements over just image inpainting with latent diffusion. \n\n**Visualization to understand how context in nearby frames help inpainting.**\nExcellent suggestion! Figure B1 in the main response PDF, visualizes how our method responds to corruptions in context frames at test time. This suggests VIDM does use info from context frames when necessary..\n\n\n**Discussion about Neural Feature Fusion Fields and NeuralDiff.**\nThanks for this pointer. This is a relevant reference that we will cite and discuss. Neural Feature Fusion Fields (like NeuralDiff) embeds inductive bias into NeRF to obtain a decomposition into a static background, transient foreground object, and agent. Using NeRF lets them infer a 3D factorization but it comes with its own limitations: a) it requires many (100s of) viewpoints to work, and b) there are no priors that can be used to complete objects that are never observed. In contrast, our method pursues a factorization in 2D, can inpaint reasonaly with just 4 frames of context, and can also use priors on appearance of objects from large-scale pre-trained diffusion models. \n\n\nWe compare to NeuralDIff (NFFF didn’t release models for EPIC videos) on P05_01 sequence since it is the only one that is common with our test set. We focus on frames that include a hand, and use their static and transient reconstruction as the prediction for $I_{env}$. We contrast it with the prediction for $I_{env}$ from our model. Figure B3 shows qualitative comparisons. On these images, our model achieves superior FID scores - 186.79 for VIDM vs 215.90 for NeuralDiff. Note that FIDs are overall higher than usual, but for good reason. There is no hand-removed image set (ie. objects floating in air) to use as reference to compute FID. As a proxy reference set, we use images that don’t contain hands as reference and thus FID scores for both models are higher than usual. \n\n**Clarification about experiments in Table 2.**\nFirst, to clarify, all methods in Table 2 return the same number of proposals. The last row, that runs the detector twice, pools together the detections from the two runs but returns the same number of detections as the baselines. Thus, in our view, the comparison, as is, is fair. One could argue that we use 2x compute time than baselines. For this, the second to last row presents a direct comparison where we run the detector just once (but on $I_{env}$) and still see improvements over the raw image and other baselines across most metrics.\n\nWe believe we have adequately addressed the concerns raised in this review. We will love to hear what you think and will be happy to offer further clarifications or respond to any other concerns. We hope our response helps improve the impression of our work."}, {"review_id": "2DNEZQ2gL4", "reviewer": "Reviewer_u9dg", "summary": "The following work presents a factorized approach for video-based egocentric tasks. Specifically, they propose to break down the video feed into separate environment-only and hands only feeds. Intuition behind this formulation is that change in appearance of the hands may constitute a domain gap when the source of the video perspective changes (e.g. person to person, person to robot manipulator). Furthermore, explicit factorization provides the model with additional supervision on the breakdown between what is the environment and what is the manipulator.\n\nThe hands are removed from the video feed using a video-inpainting model based on the latent diffusion architecture with attention-based extensions to attend to multiple past frames. Results demonstrate that their video-inpainting formulation outperforms DLFormer in both the video inpainting task (limited to their use case) as well as in improvements to downstream applications (object detection, affordance prediction, etc.)", "questions": "See weaknesses", "limitations": "Technical limitations discussed. I'm not sure it's sufficient to simply say that this work inherits uncertain societal implications from other generative modeling works.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Simple idea with extensive demonstration of improvements in multiple downstream tasks.\n- Strong results for video inpainting in the ego-centric setting with similarly very simple but pretty easy to justify design choices (optical-flow-like attention for past frames, exclusion of hands from training data)", "weaknesses": "- While the superiority of their video inpainting formulation is demonstrated only within the hand-removal task of ego-centric videos, the language used to describe the method can often be misinterpreted as a broader claim for outperforming existing  state of the art models in a general sense.\n- The authors found it helpful to exclude hands as prediction targets during training. This seems like a significant design decision that should have a corresponding ablation study.\n- Additional training details for comparison against DLFormer are missing:\n  - Was DLFormer also trained with the hand-exclusion technique?\n  - On what training data were the visual codebooks used by the LDM formulation and DLFormer derived from?\n- While I don't necessarily doubt the idea that the factorized formulation improves object detection, I do not see average recall as an appropriate replacement for average precision. I would much rather see average precision measured on a limited set of categories where all instances of the object category in question are annotated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and thoughtful feedback. Please see our response below and refer to the supporting figures in the rebuttal PDF.\n> While the superiority of their video inpainting formulation is demonstrated only within the hand-removal task of ego-centric videos, the language used to describe the method can often be misinterpreted as a broader claim for outperforming existing state of the art models in a general sense.\n\n* Thanks for the feedback. We will take a pass and further qualify our claims to be more about the hand inpainting task in egocentric videos.\n\n> The authors found it helpful to exclude hands as prediction targets during training. This seems like a significant design decision that should have a corresponding ablation study.\n\n* We are running this ablation but, unfortunately it won’t finish in the time duration of the rebuttal (the model takes about 12 days to train). Preliminary quantitative results at 3 days of training indicate that this choice is indeed effective across all metrics (PSNR of 31.14 vs 32.17 for our original model at 3 days of training. SSIM: 0.950 vs 0.955, FID 12.10 vs 10.57). However, qualitative visualizations exhibit the error mode that we saw during our development. Because this ablated model has to output hands some of the time, it sometimes paints the hand like pixels back into the image. See examples in the PDF (Figure B2) attached with the main response.\n\n> Additional training details for comparison against DLFormer are missing: Was DLFormer also trained with the hand-exclusion technique?\n\n* Yes. DLFormer is a per-clip method. It fits a unique set of model weights for each clip at test-time. Since all hands are masked out during test-time inpainting, DLFormer never sees any hands during test-time finetuning. It uses no additional pre-training step beyond having a pre-trained visual codebook. This codebook can easily reconstruct images from EPIC without hands.\n\n> On what training data were the visual codebooks used by the LDM formulation and DLFormer derived from?\n\n* Because we are finetuning pre-trained models we had to use the same codebooks that were used for the released LDM and DLFormer models: Places for LDM and COCO for DLFormer. We verified that both codebooks did a comparable job at reconstructing frames from the EPIC Dataset.\n\n\n> While I don't necessarily doubt the idea that the factorized formulation improves object detection, I do not see average recall as an appropriate replacement for average precision. I would much rather see average precision measured on a limited set of categories where all instances of the object category in question are annotated.\n\n* This is a good experiment to run. To this end we took the class with the fewest false positives (which happened to be ‘scissors’) when using raw images, and manually labeled all instances which were indeed true positives (adding missing detections to the ground truth labels). For this class with labels updated, using raw images only achieves an AP of 0.559, while using images in painted with VIDM achieves an AP of 0.584.  \n\nWe believe we have adequately addressed the concerns raised in this review. We will love to hear what you think and will be happy to offer further clarifications or respond to any other concerns. We hope our response helps improve the impression of our work."}, {"review_id": "ABMzup6PnR", "reviewer": "Reviewer_jD5i", "summary": "This work proposes the use of a factored agent and environment representation to handle two ego-centric video problems introduced by human hands : 1. They occlude objects of interaction and induce a domain gap between the data available for learning (egocentric videos) and the data seen by the robot at execution time; 2. Removing hands from the scene by masking or inpainting abandon the information of object affordances. The paper demonstrates the ability of the factored representation across tasks spanning 2D/3D visual perception to robot learning. They also show how selectively choosing and modifying aspects of the factored representation improves performance across all of these tasks compared to existing approaches.", "questions": "Please see my comment in weakness.", "limitations": "Please see my comment in weakness.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength\n1. The paper is clearly written and easy to follow. The related work provides enough information for the reviewers to get familiar with the background of ego-centric video and related tasks. \n2. The proposed diffusion model, VIDM, is effective while efficient. The performance looks amazing compared to previous work.\n3. The experiment part is sound and shows the effectiveness of the proposed VIDM across many benchmarks.\n4. The motivation is intuitive. Combining hand pose as well as inpainting technique can provide more information than before, and thus the improvement is plausible.", "weaknesses": "Weakness\n1. In figure 2, what is the meaning of the big f and g? Does they stand for different functions? If so, what is the purpose of drawing them in that way? The idea of this figure is not that clear.\n2. Which part in VIDM contributes the most to the performance improvement across those benchmark? There seems to be no related ablation study on this. \n3. In table 1, how does the stable diffusion (fine-tuned) done? It would be great if the author could provide more detail on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and thoughtful feedback. Please see our response below and refer to the supporting figures in the rebuttal PDF.\n\n> In figure 2, what is the meaning of the big f and g? Does they stand for different functions? If so, what is the purpose of drawing them in that way? The idea of this figure is not that clear.\n* Figure 2 is meant to illustrate a few of the ways one could use an agent-envorinment factorization. For example, in section a (left) we show $I_{env}$ and $I_{agent}$ being passed into a single function that optimizes some perception task. This mirrors application 1, which uses the both elements of the factorization to improve object detection. Similarly in section c (right) we show that one can independently transform the agent representation (with some function $g$) before passing into both elements into another function ($f$). This is particularly useful when working with multiple embodiments. We demonstrate this in applications 4 and 5, where we process $I_{agent}$ with a function ($g$ in the language of figure 2) that maps both robot end-effectors and human hands to the same visual representation (green dot). This processing remove the visual domain gap, improving generalization across embodiments. We will update the caption in this figure to be clearer.\n\n> Which part in VIDM contributes the most to the performance improvement across those benchmark? There seems to be no related ablation study on this.\n* In the paper we have already included an ablation, and we have added another for the rebuttal along with a diagnostic visualization as described below.\n\n  * In the paper we compare VIDM, vs Latent Diffusion finetuned on our data. This shows that there is a clear benefit to the multi-frame nature of our model, and gains aren’t just due to training on in-domain data. \n\n   * As for ablations on the nature of our training: we are currently running an ablation on the hand exclusion aspect of our training protocol (filtering out Ego4d frames with hands and not propagating loss on pixels with hands). Unfortunately it won’t finish in the time duration of the rebuttal (the model takes about 12 days to train). Preliminary quantitative results at 3 days of training indicate that this choice is indeed effective across all metrics (PSNR of 31.14 vs 32.17 for our original model at 3 days of training. SSIM: 0.950 vs 0.955, FID 12.10 vs 10.57). However, qualitative visualizations exhibit the error mode that we saw during our development. Because this ablated model has to output hands some of the time, it sometimes paints the hand like pixels back into the image. See examples in the PDF (Figure B2) attached with the main response.\n\n    *  Furthermore, in order to give some insight to how our model uses information from prior frames, we visualized how our method responds to corruptions in context frames at test time. See examples in the PDF (Figure B1) attached with the main response.\n\n> In table 1, how does the stable diffusion (fine-tuned) done? It would be great if the author could provide more detail on this.\n* We took the single frame model that we extended to multiple frames (Latent Diffusion inpainting pre-trained on Places) and simply finetuned it on the exact same data that we used for finetuning VIDM. Since the single frame model takes in no context frames it ignored the extra frames and was finetuned to inpaint the masked region in the target frame. The same hand exclusion techniques and other training choices were used for this experiment.\n\nWe believe we have adequately addressed the concerns raised in this review. We will love to hear what you think and will be happy to offer further clarifications or respond to any other concerns. We hope our response helps improve the impression of our work."}, {"review_id": "uJ0iV4b6Xb", "reviewer": "Reviewer_ymXH", "summary": "This paper proposes to use agent-environment factorization of egocentric videos to facilitate various downstream tasks (e.g., object detection, 3D reconstruction, affordance prediction, etc.). The authors leverages a pipeline to achieve agent-environment factorization. It consists of first a segmentation model that segment hands in egocentric videos, and second a diffusion-based inpainting model for filling the hand area. The benefit of the proposed pipeline is supported by improvements of baselins over various downstream tasks.", "questions": "See the Weakness section. The authors could focus on:\n1. Identify the uniqueness of the proposed VIDM and show its superiority for the current task instead of a plain pipeline.\n2. Showing that agent-environment factorization could be beneficial for interaction tasks. As this could be the most important factor for the proposed pipeline, the current evaluated domain and task might be too limited.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of agent-environment factorization is interesting and as shown by the experiments, it does facilitate downstream tasks in various scenarios.\n2. I do appreciate the sufficient amount of downstream tasks evaluated according, it proves the soundness of factorization.", "weaknesses": "1. One major concern on this paper is on its technical contribution. The proposed VIDM contains limited technical novelty as it is a basic segment-then-inpaint pipeline and does not propose any new modules.\n2. There is no video-inpainting method compared in comparative experiments (e.g., [1]). As diffusion-based models usually perform poorly in inference speed, the current comparison with image-based inpainting models do not show whether VIDM's inference speed is enough for videos.\n3. In the reward learning task (Sec. 5.5), the experiments are only conducted on 3 tasks, and all of them are open-action tasks. Would the results and analysis still hold for more complex tasks? or is it limited to the current selected domain.\n4. In the real-world policy learning task (Sec. 5.6), the action space of the robot is very limited (1D) and has been placed in a very task-specific position. It might be too simple for making a point. Additionally, these experiments were only conducted on one task (still open-action) under one secneario, this makes it difficult to assess the generalizability and effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and thoughtful feedback. Please see our response below and refer to the supporting figures in the rebuttal PDF.\n\n>One major concern on this paper is on its technical contribution. The proposed VIDM contains limited technical novelty as it is a basic segment-then-inpaint pipeline and does not propose any new modules.\n*  VIDM introduces new modules. It is more and better than just a segment-then-inpaint pipeline. Furthermore, our paper is more than just VIDM. \n\n   Specifically, VIDM uses cross-frame attention layers to transform a pre-trained image diffusion model into a video in-painter as described in Section 4 and Figure 3. We evaluate the effectiveness of this proposed architectural contribution on the video-inpainting task and observe improvements over a basic segment-then-inpaint pipeline as well as the current state-of-the-art for video inpainting. Furthermore, a major component of the paper is not just the performance of VIDM, but how we use VIDM. We propose a novel agent-environment factored representation for egocentric videos and show its effectiveness in extensive experimental evaluation across 5 benchmarks spanning 2D/3D perception to robot learning.\n\n>There is no video-inpainting method compared in comparative experiments (e.g., [1]). As diffusion-based models usually perform poorly in inference speed, the current comparison with image-based inpainting models do not show whether VIDM's inference speed is enough for videos.\n\n* Our paper already included comparisons to DLFormer, the state-of-the-art for video in-painting tasks in Table 1 and Section 5.1. We note a large improvement in metrics over this prior state-of-the-art (PSNR for DLFormer 26.98, vs 32.26 for ours). Table 1 in the DLFormer paper already reports comparison to the cut-and-paste in-painter used in [1] and reports very large improvements over cut-and-paste, thus we didn’t include a direct comparison to cut-and-paste. \n\n  Furthermore, we also report the inference speed in Table 1. VIDM needs 13.6s / image and is also much faster than 106.4s/image for DLFormer. This is not real-time but our applications don’t require real-time inference.\n\n> In the reward learning task (Sec. 5.5), the experiments are only conducted on 3 tasks, and all of them are open-action tasks. Would the results and analysis still hold for more complex tasks? or is it limited to the current selected domain.\n\n* To demonstrate that our method can work on tasks beyond opening, we followed the same protocol as in Table 4 for a fourth task of picking up a plate. In epic kitchens, there are less than ⅓ as many sequences for this task as for opening drawers, and the quality is worse (clips having the plate out of frame, annotation timing being off etc.). This lack of data and quality hurts generalization for all methods, but we still see a positive trend where using VIDM inpainted images with factorization gives Spearman’s correlation of 0.139, while raw images and non-factorized inpainting give 0.118 and 0.083 respectively. We note that many other cross embodiment learning techniques may be used with our factored representation to explore more complex or multi-stage tasks (e.g [1,7]) which we leave to future work.\n\n> In the real-world policy learning task (Sec. 5.6), the action space of the robot is very limited (1D) and has been placed in a very task-specific position. It might be too simple for making a point. Additionally, these experiments were only conducted on one task (still open-action) under one secneario, this makes it difficult to assess the generalizability and effectiveness.\n*  While this experiment may be simple, it still does make a point. Past work in this setting [1] (which is represented by the orange line in figure 5 right) doesn’t work because information of where the end-effector is relative to the object of interaction is lost. This slows down learning. AEF factorization retains the end-effector position while minimizing the domain gap and consequently learns faster. Lack of cheap dexterous manipulators limits the tasks we can tackle in the real world.\n\n   In addition to improvements over baselines in 4 other applications (2d object detection, 3d shape prediction, affordance prediction, and offline reward learning) this real world experiment demonstrates real-world feasibility of AEF+VIDM.\n\nWe believe we have adequately addressed the concerns raised in this review. We will love to hear what you think and will be happy to offer further clarifications or respond to any other concerns. We hope our response helps improve the impression of our work."}, {"review_id": "lEztbZHnUi", "reviewer": "Reviewer_t2iv", "summary": "The paper addresses challenges in using egocentric videos for robotics tasks, specifically the issues of occlusion and visual mismatch between the human hand and a robot end effector. To address these problems, this work proposes a factored representation of the scene that separates the agent (human hand) from the environment. This factorization is achieved via the proposed Video Inpainting via Diffusion Model (VIDM). Experiments demonstrate the effectiveness of VIDM in improving the inpainting quality and highlights the power of the factored representation for various downstream robotics tasks.", "questions": "1) It seems mostly $I_{t}^{\\text{env}}$ is being used; where does the agent representation come into play? In application 3, I assume you're using $I_{t}^{\\text{env}}$ to predict the GT hand pixels, so $I_{t}^{\\text{agent}}$ is unused? And in applications 4 and 5, the function $g$ which abstracts $I_{t}^{\\text{agent}}$ simply returns a green dot representing the position of the end effector/hand. This is a reasonable exploration of the concept, but are there more concrete use cases where $I_{t}^{\\text{agent}}$ obtained from segmentation models (as described in L101-103) actually interplays equally with the inpainted $I_{t}^{\\text{env}}$?\n\n2) It seems like in application 2, what the experiment is actually proving is that ground truth environment information without the agent improves 3D reconstruction (unsurprising), and not \"the effectiveness of our proposal of using factorized agent-environment representations\" (L228-229). In other words, there's no connection at all between this experiment and the factorized representation that you learn from VIDM.\n\n3) In applications 4 and 5, neither of these tasks lend themselves to needing this agent-environment factorized representation. In particular, I feel that in opening a drawer, cupboard or fridge, the occlusion by the hand/end-effector does not significantly hamper the accomplishing of the task. Even in Figure 5, I see that the (salient) occlusion by the green dot in the agent-agnostic representation is almost more extreme than the occlusion by the robotic end-effector in the raw image. What's the intuition here, or could it simply be the case that the baselines were not tuned sufficiently? I ask because in Table 4, the \"inpainted only\" performance is much better than the \"raw image\" performance, but in Figure 5 for the real-world experiments, \"inpainted only\" actually fails entirely compared to \"raw image.\" Can the authors provide some explanation for this?", "limitations": "Limitations are addressed well in the discussion section of the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1) Paper is well-written and well-structured; the descriptions of the experimental protocols for each of the applications was very helpful.\n\n2) Experiments are very comprehensive; I was happy to see evaluation on both the inpainting quality achieved by the proposed model, as well as the utility of its associated representations for various downstream robotics tasks.\n\n", "weaknesses": "1) The main weakness that stands out to me is that the current set of downstream robotics applications are not as convincing as they need to be. Specifically, in many of the tasks, it seems the inpainted environment $I_{t}^{\\text{env}}$ is the key component, and not the factorized representation in its entirety.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and thoughtful feedback. Please see our response below and refer to the supporting figures in the rebuttal PDF.\n\n> It seems mostly Ienv is being used; where does the agent representation come into play? In application 3, I assume you're using Ienv to predict the GT hand pixels, so Iagent  is unused? And in applications 4 and 5, the function g which abstracts Iagent  simply returns a green dot representing the position of the end effector/hand. This is a reasonable exploration of the concept, but are there more concrete use cases where Iagent obtained from segmentation models (as described in L101-103) actually interplays equally with the inpainted env?\n* To clarify: when making predictions for application 3, $I_{agent}$ is not used as an input to the model, $f$. However in the parlance of figure 2, aspects of $I_{agent}$ (the pose of the hand) is used to provide the supervision to train $f$ to predict grasps afforded by objects (GAO task) in Table 3 and Section 5.4.\n\n   More generally, many past works re-target human hand pose to robot pose (eg [DexMV], [DexVIP], [Robotic Telekinesis] among others) for tele-operation and imitation learning. These applications involve using $I_{agent}$ to infer hand pose (and not just grasp type as in our work). One could also imagine a content creation application where $I_{agent}$ is used to predict aspects of $I_{env}$ (what object would fit into an animated character’s hand, for instance). Thus, we believe there are many other applications of our proposed factorization (and specifically $I_{agent}$)  than what we were able to experiment with in our paper.\n\n> It seems like in application 2, what the experiment is actually proving is that ground truth environment information without the agent improves 3D reconstruction (unsurprising), and not \"the effectiveness of our proposal of using factorized agent-environment representations\" (L228-229). In other words, there's no connection at all between this experiment and the factorized representation that you learn from VIDM.\n* Your understanding of application 2 is correct. However, the ground truth is exactly the ground truth version of $I_{env}$ in our proposed agent-environment factorization that our model tries to approximate. This experiment was designed to independently test the efficacy of AEF from VIDM, as such we use ground truth factorized images.\n\n> In applications 4 and 5, neither of these tasks lend themselves to needing this agent-environment factorized representation. In particular, I feel that in opening a drawer, cupboard or fridge, the occlusion by the hand/end-effector does not significantly hamper the accomplishing of the task. Even in Figure 5, I see that the (salient) occlusion by the green dot in the agent-agnostic representation is almost more extreme than the occlusion by the robotic end-effector in the raw image. What's the intuition here, or could it simply be the case that the baselines were not tuned sufficiently? I ask because in Table 4, the \"inpainted only\" performance is much better than the \"raw image\" performance, but in Figure 5 for the real-world experiments, \"inpainted only\" actually fails entirely compared to \"raw image.\" Can the authors provide some explanation for this?\n\n* AEF offers a solution for both occlusion and the domain gap between human hands and the robot end-effector. In applications 4 and 5, the difficulty is not  so much about occlusion, but rather about the domain gap between human hands and the robot end effector. This domain gap is mitigated by the use of the same green dot across embodiments.\n\n   Excellent observation about Table 4 and Figure 5!  All methods learn to correctly score frames after the objects have been manipulated. But they have different behavior while the hand/robot end-effector is approaching the object. In Table 4, the camera motion in the egocentric claw videos provides a good signal for approaching the goal (thus high Spearman’s rho), while the robot experiment in Figure 5 has a fixed camera. The fixed camera in the robot experiment means that the reward function in the “Inpainted only” baseline doesn’t provide any feedback on the task progress, whereas our method, with its green dot, does. Hopefully, this clarifies your concern.\n\nWe believe we have adequately addressed the concerns raised in this review. We will love to hear what you think and will be happy to offer further clarifications or respond to any other concerns. We hope our response helps improve the impression of our work.\n\n**References:**\n\n[DexMV] DexMV: Imitation Learning for Dexterous Manipulation from Human Videos. ECCV 2022.\n\n[DexVIP] DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video. CoRL 2021.\n\n[Robotic Telekinesis] Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on YouTube. RSS 2022."}]

Author rebuttal:
We would like to thank all reviewers for their valuable and insightful comments. Attached to this post is a single page pdf containing 3 figures: B1, B2, and B3. B1 is a diagnostic visualization showcasing VIDM’s ability to intelligently copy pixels from previous frames. B2 visualizes failure modes of an ablation of our method that allows loss to propagate to pixels containing hands. B3 compares reconstructions from VIDM against those from NeuralDiff, demonstrating VIDM’s superior ability to recover occluded pixels. We have posted replies to each reviewer's individual comments to address their specific concerns. 

Meta-review: The reviewers found this work on factored scene representation to be making a valuable contribution despite some shortcomings in the evaluation. The AC thinks it is likely to give rise to similar methods useful for pixel-based decision-making scenarios beyond robotics. This work is recommended for acceptance, provided the authors incorporate their fixes to the issues discussed during the rebuttal stage into the camera-ready version.
Decision: Accept (poster)