SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results. Tab. 2 reports the average recall when using different images as input to the COCO-trained Mask RCNN detector. We compare against a) just using the raw image (i.e.  $I_{t}$ ) as would typically be done, b) using a masked image (i.e.  $I_{t}$  with  $m_{t}^{\mathrm{agent}}$  blacked out) a naive way to remove the hand, c) different methods for inpainting the agent pixels (pre-trained and finetuned single-frame Latent Diffusion Model [63] and our inpainting scheme, VIDM), and d) using both the original image and the inpainted image (i.e. both  $I_{t}$  and  $I_{t}^{\mathrm{env}}$ ) as enabled by AEF. When using both images, we run the pre-trained detector on each image, merge the predictions, and return the top- $k$ .
Naive masking introduces artifacts and hurts performance. Inpainting using a single-frame model helps but not consistently. Our video-based inpainter leads to more consistent gains over the raw image baseline, outperforming other inpainting methods. Our full formulation leads to the strongest performance achieving a  $24\% - 51\%$  relative improvement over the raw image baseline.
5.3 Application 2: 3D Reconstruction of Hand-Held Objects
Past work has tackled the problem of 3D reconstruction of hand-held objects [22, 34, 59, 95]. Here again, the human hand creates occlusion and hence nuisance, yet it provides valuable cues for the object shape [95]. Similar to Sec. 5.2, we hypothesize that the complete appearance of the object, behind the hand, may provide more signal to a 3D reconstruction model.
Protocol. We adopt the state-of-the-art approach from Ye et al. [95]. They design a custom neural network architecture that is trained with full supervision on the ObMan dataset [22]. Their model just accepts  $I_{t}$  as input. To test whether the complete appearance of the object is helpful, we additionally input  $I_{t}^{\mathrm{env}}$  to their model. As ObMan is a synthetic dataset, we use ground-truth  $I_{t}^{\mathrm{env}}$  images obtained by rendering the object without the hand. This evaluation thus only measures the impact of the proposed factorization. We use the standard metrics: F1 score at 5 & 10 mm, and chamfer distance.
Results. Using  $I_{t}$  and ground truth  $I_{t}^{\mathrm{env}}$  improves upon just using the raw image (as used in [95]) across all metrics: the F1 score at 5mm increases from 0.41 to 0.48, the F1 score at 10mm increases from 0.62 to 0.68, and the chamfer distance improves from 1.23 to 1.06. As we are using ground truth  $I_{t}^{\mathrm{env}}$  here, this is not surprising. But, it does show the effectiveness of our proposal of using factorized agent-environment representations.
5.4 Application 3: Affordance Prediction
Past works [19] have used videos to learn models for labeling images with regions of interaction and afforded grasps. While egocentric videos directly show both these aspects, learning is made challenging because a) the image already contains the answer (the location and grasp type exhibited by the hand), and b) the image patch for which the prediction needs to be made is occluded by the hand. Goyal et al. [19] mask out the hand and train models to predict the hand pixels from the surrounding context. This addresses the first problem but makes the second problem worse. After masking there is even more occlusion. This application can be tackled using our AEF framework, where we use inpainted images  $I_{t}^{\mathrm{env}}$  to predict aspects of the agent shown in  $I_{t}^{\mathrm{agent}}$ .
Table 4: Spearman's rank correlations of reward functions learned on Epic-Kitchens [12] when evaluated on robotic gripper sequences for opening drawer, cupboard, and fridge tasks. Our factored representation achieves better performance than raw images or environment-only representation.
Input Representation	Drawer	Cupboard	Fridge	Overall
Raw Images (i.e. It)	0.507	0.507	0.660	0.558
Inpainted only (i.e. Itenv as proposed in [1])	0.574	0.570	0.671	0.605
AEF (Itenv and g(Itagent)) (Ours)	0.585	0.582	0.676	0.614
Protocol. We adopt the training scheme from [19] (current state-of-the-art on this benchmark), but instead of using masked images as input to the model, we use images inpainted using our VIDM model. We adopt the same metrics and testing data as used in [19] and report performance on both the Region of Interaction (RoI) and Grasps Afforded by Objects (GAO) tasks. We test in a small data regime and only use 1000 images for training the affordance prediction models from [19]. We report mean and standard deviation across 3 trainings.
Results. Tab. 3 presents the metrics. Across both tasks, the use of our inpainted images leads to improved performance vs. using masked-out images.
5.5 Application 4: Learning Reward Functions from Videos
Difficulty in manually specifying reward functions for policy learning in the real world has motivated previous work to learn reward functions from human video demonstrations [1,6,9,71]. The visual mismatch between the human hand and robot end-effector is an issue. Past work [1] employs inpainting to circumvent this but consequently loses the important signal that the hand motion provides. Our factored representation in AEF provides a more complete solution and, as we will show, leads to improved performance over just using the raw data [9, 71] and inpainting out the hand [1].
Protocol. We train a reward predictor on video clips from Epic-Kitchens [12] and assess the quality of the reward predictor on trajectories from a robotic end-effector. Specifically, we consider three tasks: opening a drawer, opening a cabinet, and opening a refrigerator. We use the action annotations from [12] to find video segments showing these tasks to obtain 348, 419, and 261 clips respectively. The reward function is trained to predict the task progress (0 at clip start and 1 at clip end) from a single image. The learned function is used to rank frames from trajectories of a robotic end-effector, specifically the reacher-grabber assistive tool from [76, 98] captured using a head-mounted GoPro HERO7. We collected 10, 10, and 5 trajectories respectively for drawers, cupboards, and refrigerators. These robotic trajectories represent a large visual domain gap from the human-handed training demonstrations and test how well the learned reward functions generalize. Following [7], we measure Spearman's rank correlation between the rewards predicted by the learned reward function and the ground truth reward (using frame ordering) on the pseudo-robotic demonstrations. The Spearman's rank correlation only measures the relative ordering and ignores the absolute magnitude.
Results. Tab. 4 reports the Spearman's rank correlations for the different tasks. We compare the use of different representations for learning these reward functions: a) raw images (i.e.  $I_{t}$ ), b) just inpainting (i.e.  $I_{t}^{\mathrm{env}}$ ), and c) our factored representation (i.e.  $I_{t}^{\mathrm{env}}$  and  $I_{t}^{\mathrm{agent}}$ ). For our factored representation, we map  $I_{t}^{\mathrm{agent}}$  through a function  $g$  to an agent-agnostic representation by simply placing a green dot at the highest point on the end-effector mask (human hand or robot gripper) in the image plane (see Fig. 5), thus retaining information about where the hand is in relation to the scene. Our factored model, AEF, correlates the best with ground truth. Just painting out the end-effector loses important information about where the end-effector is in relation to objects in the scene. Raw frames maintain this information but suffer from the visual domain gap.
5.6 Application 5: Real-World Policy Learning using Learned Rewards
Motivated by the success of the offline evaluations in Sec. 5.5, we use the learned reward function for opening drawers to learn a drawer opening policy in the real world using a Stretch RE2 robot (see Fig. 5 (left)).
Figure 5: Real-world experiment setup and results. (left) Raw views from camera, (center) Inpainted image with agent-agnostic representation (green dot at top-pixel of end-effector). (right) Success rate as a function of CEM iterations.
Protocol. We position the robot base at a fixed distance from the target drawer. We use a 1D action space that specifies how far from the base the hand should extend to execute a grasp, before retracting. The drawer opens if the robot is able to correctly grasp the handle. We use a GoPro camera to capture RGB images for reward computation. The total reward for each trial is the sum of the predicted reward at the point of attempted grasp, and after the arm has been retracted. We use the Cross-Entropy Method (CEM) [14] to train the policy. We collect 20 trials in each CEM iteration and use the top-7 as elite samples. Actions for the next CEM iteration are sampled from a Gaussian fit over these elite samples. We use the success rate at the different CEM iterations as the metric.
Results. Similar to Sec. 5.5, we compare against a) raw images (i.e.  $I_{t}$ ), b) inpainted images (i.e. just  $I_{t}^{\mathrm{env}}$  as per proposal in [1]), and c) our factored representation, AEF. As before, we use a dot at the end-effector location as the agent-agnostic representation of  $I_{t}^{\mathrm{agent}}$ . We use our VIDM for inpainting images for all methods. We finetune a DeepLabV3 [9] semantic segmentation model (on 100 manually annotated images) to segment out the Stretch RE2 end-effector. Fig. 5 (right) shows the success rate as a function of CEM iterations. Reward functions learned using our AEF representation outperform baselines that just use the raw image or don't use the factorization.
Our experiments have demonstrated the effectiveness of our inpainting model. Specifically, the use of strong priors from large-scale pre-trained inpainting diffusion models and the ability to steal content from previous frames allows VIDM to outperform past methods that only use one or the other cue (pre-trained LDM [63] and DLFormer [62] respectively). This is reflected in qualitative metrics in Tab. 1 and also in qualitative visualizations in Fig. 4.
Our powerful video inpainting model (along with semantic segmentation models that can segment out the agent) enables the extraction of our proposed Agent-Environment Factored (AEF) representation from egocentric video frames in pixel space. This allows intuitive and independent manipulation of the agent and environment representations, enabling a number of downstream visual perception and robotics applications. Specifically, we demonstrated results on 5 benchmarks spanning 3 categories. For visual perception tasks (object detection and 3D reconstruction of hand-held objects), we found additionally using the environment representation, which has the agent removed, improved performance. For affordance prediction tasks (region of interaction and grasps afforded by objects), we found that using the inpainted environment representation to be more effective at predicting relevant aspects of the agent representation than naive masking of the agent as used in past work. For robotic tasks (learning rewards from videos), the factored representation allowed easy conversion of the agent representation into an agent-agnostic form. This led to better transfer of reward functions across embodiments than practices adopted in past work (using the original agent representation and ignoring the agent altogether) both in offline evaluations and online learning on a physical platform.
Given that all information comes from the masked input frames, one might wonder, what additional value does the proposed inpainting add? First, it provides the practical advantage that each downstream application doesn't need to separately pay the implicit training cost for interpreting
masked-out images. Inpainted images are like real images, allowing ready re-use of the off-the-shelf models. Furthermore, the data complexity of learning a concept under heavy occlusion may be much higher than without. A foundation inpainting model can leverage pre-training on large-scale datasets, to inpaint the occlusion. This may be a more data-efficient way to learn concepts as our experiments have also shown.
7 Limitations and Broader Impact
As with most work with diffusion models, inference time for our method is 13.6 seconds which renders real-time use in robotic applications infeasible. AEF only inpaints out the hand to recover the unoccluded environment. There may be situations where the environment might occlude the agent and it may be useful to explore if similar inpainting could be done for agent occlusion.
At a broader level, while our use of the video inpainting model was for robotic applications, the VIDM model can also be thought of a general-purpose video inpainting model. This inherits the uncertain societal implications of generative AI models.
We thank Ruisen Tu and Advait Patel for their help with collecting segmentation annotations for the claw and the robot end-effector. We also thank Arjun Gupta, Erin Zhang, Shaowei Liu, and Joseph Lubars for their feedback on the manuscript. This material is based upon work supported by NSF (IIS2007035), DARPA (Machine Common Sense program), NASA (80NSSC21K1030), an Amazon Research Award, an NVidia Academic Hardware Grant, and the NCSA Delta System (supported by NSF OCI 2005572 and the State of Illinois).
[1] Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human-to-robot imitation in the wild. In RSS, 2022. 1, 2, 3, 8, 9, 17
[2] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 2
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. edifi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv:2211.01324, 2022. 3
[4] Marcelo Bertalmio, Lunita Vese, Guillermo Sapiro, and Stanley Osher. Simultaneous structure and texture image inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2003. 3
[5] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. arXiv, 1512.03012, 2015. 16
[6] Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos. In NeurIPS, 2020. 8
[7] Matthew Chang, Arjun Gupta, and Saurabh Gupta. Learning value functions from undirected state-only experience. In International Conference on Learning Representations, 2022. 8, 17
[8] Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from "in-the-wild" human videos. arXiv preprint arXiv:2103.16817, 2021. 3
[9] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017. 4, 8, 9
[10] Enric Corona, Albert Pumarola, Guillem Alenyà, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 16
[11] Antonio Criminisi, Patrick Perez, and Kentaro Toyama. Object removal by exemplar-based inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2003. 3
S1 VIDM Training Details
For an overview of the VIDM model architecture, see Section 4. In each block, the CNN layers are implemented as residual blocks [25] with SiLU non-linearities [26] and each attention layer does self-attention across all token from all input images using 32-channel GroupNorm normalization. Following [63], upsampling and downsampling operations are both implemented using residual CNN blocks with either an internal nearest mode  $2 \times$  upsampling operation or internal  $2 \times$  downsampling via average pooling. An initial convolution brings the feature dimension to 256, which is raised to a maximum of 1024 at the center of the U-Net. At the highest spatial resolution of  $64 \times 64$  the self-attention layer is omitted, as attention with  $16384 (= 64 \times 64 \times 4)$  tokens is computationally intractable for our available hardware. The largest attention layer occurs at a spatial resolution of  $32 \times 32$  across four images for a total of 4096 tokens.
We trained VIDM using target images from Ego4D [20] and VISOR [13] (see Section 4). Since no evaluation was done on Ego4D, no Ego4D data was held out. For VISOR, all data from participants
P37, P35, P29, P05, and P07 was held-out from training. This held-out data from these participants was used for reconstruction quality evaluation (Section 5.1) and object detection (Section 5.2) experiments. Table S1 lists hyper-parameters. Figure S3 shows sample training batches.
Table S1: VIDM Model and Training Hyper-parameters.
Hyper-parameter	Value
Learning Rate	4.8 × 10-5
Batch Size	48
Optimizer	Adam
Diffusion Steps (training)	1000
Latent image Size	64 × 64
Number of VQ Embedding Tokens	8192
VQ Embedding Dimension	3
Diffusion Steps (inference)	200
Attention Heads	8
S2 Downstream Task Experimental Details
S2.1 Detection
We used off-the-shelf Mask R-CNN R_101_FPN_3x from Detectron2 [24, 89] trained on the COCO dataset [39] for evaluation. We used overlapping classes between the VISOR [13] annotations and COCO for evaluation. These were: apple, banana, bottle, bowl, broccoli, cake, carrot, chair, cup, fork, knife, microwave, oven, pizza, refrigerator, sandwich, scissors, sink, spoon, toaster.
S2.2 Affordance Prediction
Dataset: We experiment on EPIC-ROI and GAO tasks from Goyal et al. [19]. EPIC-ROI uses the EPIC-KITCHENS dataset [12] and GAO uses YCB-Affordance [10] dataset. We consider a low data regime in our work and sample  $1K$  images from these datasets to train the different models. For EPIC-ROI, we sample images with a probability inversely proportional to the length of the video. For GAO, we sample randomly. We use the same evaluation setting from [19].
Model: We use the same architecture from ACP [19] and replace the EPIC-ROI input images with images produced by our inpainting model (with hands removed) to incorporate our factorized representation. While ACP [19] masks out a patch at the bottom center of the image to hide the hand, we do not need any mask (neither for training nor for testing) since the hands have been removed via inpainting. The input is processed by ResNet-50 followed by different decoders for EPIC-ROI and GAO tasks.
Training: We train separate models for EPIC-ROI and GAO using the loss function and hyperparameters from ACP [19]. While it is possible to train a single model in multitask manner, we observe that the two tasks are not complementary to each other. We train using 3 seeds for each task and report the mean and standard deviation in the metrics.
S2.3 3D Reconstruction of Hand-held Objects
Dataset: We use ObMan [22] dataset which consists of  $2.5K$  synthetic objects from ShapeNet [5]. We use the train and test splits provided by Ye et al. [95]. We divide the train split into train and val set. The train set consists of  $134K$ , val set  $7K$  and test set  $6.2K$  images. The dataset provides 3D CAD models for each object, which we use for training hand-held object reconstruction model from Ye et al. [95].
Figure S1: The environment factorization obtained by NeuralDiff contains artifacts, e.g. (left) region around banana, (right) partial hand. Our VIDM model produces more realistic results.
Model: We use the architecture from Ye et al. [95]. It uses FrankMocap [65] to extract hand articulation features from a single image using MANO [64] hand parameterization. These hand features are used as conditioning to a DeepSDF [56] model which predicts the object shape using implicit representation. This model also takes in pixel-aligned features and global image features along with hand features. To incorporate our factorized representation, we also extract global image features and pixel-aligned features from ObMan images showing only objects (with hands removed). These features are concatenated with the features from the input ObMan images and fed as input to the DeepSDF [56] decoder.
Training: Following [95], we use a normalized hand coordinate frame for sampling points and predicting SDFs. We sample 8192 points in  $[-1,1]^3$  for training, out of which half of them lie inside and the rest lie outside the object. At test time,  $64^3$  points are sampled uniformly in  $[-1,1]^3$ . We train the model in a supervised manner using 3D ground truth from ObMan [22] for 200 epochs with a learning rate of  $1e - 5$ . Other hyper-parameters are used directly from [95].
S3.1 NERF Comparison
Neural radiance fields [49] represent an alternative path to producing agent-environment factorizations. We compare against NeuralDiff [82] as a representative method. We can only compare on the EPIC-Kitchens P05_01 sequence since it is the only one model released from that project that is common with our test set. We focus on frames that include a hand, and use their static and transient reconstruction as the prediction for  $I^{\mathrm{env}}$ . We contrast it with the prediction for from our model.
Figure S1 shows qualitative comparisons. On these images, our model achieves superior FID scores - 186.79 for VIDM vs 215.90 for NeuralDiff. Note that FIDs are overall higher than usual, but for good reason. There is no hand-removed image set (ie. objects floating in air) to use as reference to compute FID. As a proxy reference set, we use images that don't contain hands as reference and thus FID scores for both models are higher than usual.
S3.2 Robot Learning: Pick-Up-Plate
To demonstrate that our method can work on tasks beyond opening, we followed the same protocol as in Table 4 for a fourth task of picking up a plate. In epic kitchens, there are less than  $\frac{1}{3}$  as many sequences for this task as for opening drawers, and the quality is worse (clips having the plate out of frame, annotation timing being off etc.). This lack of data and low quality hurts generalization for all methods, but we still see a positive trend where using VIDM inpainted images with factorization gives Spearman's correlation of 0.139, while raw images and non-factorized inpainting give 0.118 and 0.083 respectively. We note that many other cross embodiment learning techniques may be used with our factored representation to explore more complex or multi-stage tasks (e.g [1,7]) which we leave to future work.
S3.3 Average Precision for Object Detection
In addition to the recall results reported in Table 2 we performed an small experiment to access our methods effect on object detection precision. To this end we took the class with the fewest
$I_{t - 3}$
$I_{t - 2}$
$I_{t - 1}$
$I_{t}$
$I_{t}$  in-painted (output)
Figure S2: When the oranges are occluded by the hand, but visible in the previous frame, VIDM is able to reconstruct the oranges using this information (first row). When the oranges in previous frames are covered by either black or red masks, the model fails to reconstruct the oranges and inpaints red or black pixels instead (second and third rows). This suggests that the model does refer to content from the context frames for in-painting.
false positives (which happened to be 'scissors') when using raw images, and manually labeled all instances which were indeed true positives (adding missing detections to the ground truth labels). For this class with labels updated, using raw images only achieves an AP of 0.738, while using images in painted with VIDM achieves an AP of 0.762.
S4 Visualizations
In Figure S3, we include a visualization of a training batch for our method, showcasing supervision and generated masks. In Figure S4, we include additional visualizations of the predictions made by our method and baselines.