SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Matthew Chang Aditya Prakash Saurabh Gupta
University of Illinois, Urbana-Champaign
{mc48, adityap9, saurabhg}@illinois.edu
The analysis and use of egocentric videos for robotic tasks is made challenging by occlusion due to the hand and the visual mismatch between the human hand and a robot end-effector. In this sense, the human hand presents a nuisance. However, often hands also provide a valuable signal, e.g. the hand pose may suggest what kind of object is being held. In this work, we propose to extract a factored representation of the scene that separates the agent (human hand) and the environment. This alleviates both occlusion and mismatch while preserving the signal, thereby easing the design of models for downstream robotics tasks. At the heart of this factorization is our proposed Video Inpainting via Diffusion Model (VIDM) that leverages both a prior on real-world images (through a large-scale pre-trained diffusion model) and the appearance of the object in earlier frames of the video (through attention). Our experiments demonstrate the effectiveness of VIDM at improving inpainting quality on egocentric videos and the power of our factored representation for numerous tasks: object detection, 3D reconstruction of manipulated objects, and learning of reward functions, policies, and affordances from videos.
Observations of humans interacting with their environments, as in egocentric video datasets [12, 20], hold the potential to scale up robotic policy learning. Such videos offer the possibility of learning affordances [2, 19], reward functions [1] and object trajectories [57]. However, a key bottleneck in these applications is the mismatch in the visual appearance of the robot and human hand, and the occlusion caused by the hand.
Human hands can often be a nuisance. They occlude objects of interaction and induce a domain gap between the data available for learning (egocentric videos) and the data seen by the robot at execution time. Consequently, past work has focused on removing hands from the scene by masking [19] or inpainting [1]. However, hands also provide a valuable signal for learning. The hand pose may reveal object affordances, and the approach of the hand toward objects can define dense reward functions for learning policies.
In this work, we propose the use of a factored agent and environment representation. The agent representation is obtained by segmenting out the hand, while the environment representation is obtained by inpainting the hand out of the image (Fig. 1). We argue that such a factored representation removes the nuisance, but at the same time preserves the signal for learning. Furthermore, the factorization allows independent manipulation of the representation as necessary. E.g., the agent representation could be converted to a form that is agnostic to the embodiment for better transfer across agents. This enables applications in 2D/3D visual perception and robot learning (see Fig. 2).
Project website with code, video, and models: https://matthewchang.github.io/vidm.
Figure 1: Agent-Environment Factorization of Egocentric Videos. Occlusion and the visual mismatch between humans and robots, make it difficult to use egocentric videos for robotic tasks. We propose a pixel-space factorization of egocentric videos into agent and environment representations (AEF, Sec. 3). An agent representation  $I_{t}^{\mathrm{agent}}$  is obtained using a model to segment out the agent. The environment representation  $I_{t}^{\mathrm{env}}$  is obtained by inpainting out the agent from the original image using VIDM, a novel Video Inpainting Diffusion Model (Sec. 4). AEF enables many different visual perception and robotics tasks (Fig. 2 and Sec. 5).
Figure 2: Agent-Environment Factored (AEF) representations enable many applications. (a) For visual perception tasks (Sec. 5.2 and 5.3), the unoccluded environment in  $I_{t}^{\mathrm{env}}$  can be used in addition to the original image. (b) For affordance learning tasks (Sec. 5.4), the unoccluded environment  $I_{t}^{\mathrm{env}}$  can be used to predict relevant desirable aspects of the agent in  $I_{t}^{\mathrm{agent}}$ . (c) For reward learning tasks (Sec. 5.5 and 5.6) agent representations can be transformed into agent-agnostic formats for more effective transfer across embodiments.
But how do we obtain such a factored representation from raw egocentric videos? While detection & segmentation of hands are well-studied [13, 70, 103], our focus is on inpainting. Here, rather than just relying on a generic prior over images, we observe that the past frames may already have revealed the true appearance of the scene occluded by the hand in the current time-step. We develop a video inpainting model that leverages both these cues. We use a large-scale pre-trained diffusion model for the former and an attention-based lookup of information from the past frames for the latter. We refer to it as Video Inpainting via Diffusion (VIDM). Our approach outperforms DLFormer [62], the previous state-of-the-art for video inpainting, by a large margin (Tab. 1) and is  $8 \times$  faster at test time.
Next, we demonstrate the ability of the factored representation across tasks spanning 2D/3D visual perception to robot learning. Specifically, we adopt 5 existing benchmark tasks: a) 2D detection of objects of interaction [13], b) 3D reconstruction of hand-held objects [59, 95], c) learning affordances (where to interact and how) from egocentric videos [19], d) learning reward functions, and e) their use for interactive policy learning [1]. We show how selectively choosing and modifying aspects of the factored representation improves performance across all of these tasks compared to existing approaches. We believe the advances presented in this paper will enable the use of egocentric videos for learning policies for robots.
Robot Learning from Hand-Object Interaction Data. Many past works have sought to use human-object interaction data for robotic tasks. Researchers have used videos to predict: regions of interactions [19,50], grasps / hand pose afforded by objects [19,96], and post-grasp trajectories [2,42].
Figure 3: Video Inpainting Diffusion Models (VIDM). We extend pre-trained single-frame inpainting diffusion models [63] to videos. Features from context frames  $(I_{t - h},\dots ,I_{t - 1})$  are introduced as additional inputs into the Attention Block  $A$ . We repeat the multi-frame attention block 8 times (4 to encode and 4 to decode) to construct the U-Net [66] that conducts 1 step of denoising. The U-Net operates in the VQ encoder latent space [63].
[59, 95] predict 3D shapes for hand-held objects. [1, 8, 69] learn reward functions from videos for learning real-world policies. Others use human-object interaction videos to pre-train feature representations for robotic policies [51, 60, 91]. While some papers ignore the gap between the hand and the robot end-effector [51, 60, 91], others adopt ways to be agnostic to the hand by masking it out [19], inpainting it [1], converting human pixels to robot pixels [73, 92], or learn embodiment invariant representations [8, 69, 100]. Instead, we pursue a factored representation that allows downstream applications to choose how to use the agent and the environment representations. Different from [103], we remove only the hand and retain the object of interaction.
Diffusion Models have been successful at unconditional [28, 74, 77] and conditional [15, 55, 63] image synthesis for several applications, e.g. text-to-image generation [3, 54, 61], image editing [27, 43, 48], video generation [21, 30, 86], video prediction and infilling [31] (similar to video inpainting, but focused on generating entire frames), 3D shape synthesis [45, 101, 106], self-driving applications [72, 107]. While diffusion models produce impressive results, they are computationally expensive due to the iterative nature of the generative process. To mitigate this limitation, several improvements have been proposed, e.g. distillation [44, 68], faster sampling [16, 75, 88], cascaded [29] and latent [63, 83] diffusion models. Among these, latent diffusion models (LDM) are the most commonly used variant. LDMs use an autoencoder to compress the high-dimensional image inputs to a lower-dimension latent space and train diffusion models in this latent space. In our work, we build off LDMs.
Inpainting requires reasoning about the local and global structure of the image to perform image synthesis. While earlier approaches focus on preserving local structure [4, 11, 17, 23], they are unable to capture complex geometries or large missing areas. Recent works alleviate these issues using neural networks [38, 40, 47, 58] to incorporate large receptive fields [46, 80], intermediate representations (e.g. edges [52, 93], segmentation maps [78]), adversarial learning [33, 78, 97, 99, 104]. With advancements in generative modeling, diffusion models have also been very effective on this task [43, 54, 63, 67, 96]. These approaches operate by learning expressive generative priors. An alternative is to utilize videos [36, 41, 62, 90] to reason about temporal consistency [32, 53, 87], leverage optical flow [79, 94], utilize neural radiance fields [81, 82], or extract context about the missing parts from the previous frames [35-37, 87]. Recent video-based inpainting methods incorporate transformers [41, 62, 102], cycle-consistency [90] & discrete latent space [62] to achieve state-of-the-art results. While existing approaches inpaint the entire video, we focus on inpainting a specific frame (previous frames are used as context) as required by downstream tasks. Our work adopts diffusion models for video-based inpainting to learn factored agent-environment representations, specifically, building on the single-frame inpainting model from [63].
3 Agent-Environment Factored (AEF) Representations
Motivated by the recent success of generative models, we develop our factorization directly in the pixel space. Given an image  $I_{t}$  from an egocentric video, our factored representation decomposes
it into  $I_{t}^{\mathrm{agent}}$  and  $I_{t}^{\mathrm{env}}$ . Here,  $I_{t}^{\mathrm{env}}$  shows the environment without the agent, while  $I_{t}^{\mathrm{agent}}$  shows the agent (Fig. 1) without the environment. These two images together constitute our Agent-Environment Factored (AEF) Representation.
Using the Factorization. This factorization enables the independent use of agent / environment information as applicable. For example, when hands are a source of nuisance (e.g. when object detectors are pre-trained on non-egocentric data without hands in Fig. 2a), we can use  $I_{t}^{\mathrm{env}}$  in addition to  $I_{t}$  to get a better view of the scene. In other situations, it may be desirable to predict agent properties afforded by different parts of the environments (Fig. 2b). In such a situation,  $I_{t}^{\mathrm{env}}$  can be used to predict the necessary aspects of  $I_{t}^{\mathrm{agent}}$ . In yet other situations, while the location of the agent provides useful information, its appearance may be a nuisance (e.g. when learning dense reward functions from egocentric videos for finetuning policies for robots in Fig. 2c). In such a situation, models could be trained on  $I_{t}^{\mathrm{env}}$  and  $g(I_{t}^{\mathrm{agent}})$ , where the function  $g$  generates the necessary abstractions of the agent image.
Extracting the Factorization from Egocentric Videos. For  $I_{t}^{\mathrm{agent}}$ , we employ segmentation models to produce masks for the agent in  $I_{t}$ . For egocentric videos, we use the state-of-the-art VISOR models [13] to segment out the hand. When the depicted agent is a robot, we use DeepLabV3 [9] pretrained on MS-COCO [39] and fine-tuned using manually annotated robot end-effector frames. Robot end-effectors have a distinctive appearance and limited variety, hence we can train a high-performing model with only a small amount of training data. We denote the agent segmentation in  $I_{t}$  by  $m_{t}^{\mathrm{agent}}$ .
Extracting  $I_{t}^{\mathrm{env}}$  is more challenging because of significant scene occlusion induced by the agent in a given frame. Naively extracting  $I_{t}^{\mathrm{env}}$  by just masking out the agent leaves artifacts in the image. Thus, we inpaint the image to obtain  $I_{t}^{\mathrm{env}}$ . We condition this inpainting on the current frame  $I_{t}$  and previous frames from the video, i.e.  $\{I_{t - h},\dots ,I_{t - 1}\}$  since occluded parts of the scene in  $I_{t}$  may actually be visible in earlier frames. This simplifies the inpainting model, which can steal pixels from earlier frames rather than only relying on a generic generative image prior. We denote the inpainting function as  $p(I_t,m_t^{\mathrm{agent}},\{m_{t - h}^{\mathrm{agent}},\ldots ,m_{t - 1}^{\mathrm{agent}}\} ,\{I_{t - h},\ldots ,I_{t - 1}\})$ , and describe it next.
4 Video Inpainting via Diffusion Models (VIDM)
The inpainting function  $p(I_t, m_t^{\mathrm{agent}}, \{m_{t-h}^{\mathrm{agent}}, \dots, m_{t-1}^{\mathrm{agent}}\}, \{I_{t-h}, \dots, I_{t-1}\})$  inpaints the mask  $m_t^{\mathrm{agent}}$  in image  $I_t$  using information in images  $I_{t-h}$  through  $I_t$ . The function  $p$  is realized through a neural model that uses attention [85] to extract information from the previous frames. We train this neural network using latent diffusion [63], which employs denoising diffusion [28] in the latent space obtained from a pre-trained vector quantization (VQ) encoder [18, 84].
Model Architecture. We follow prior work [63] and train a model  $\epsilon_{\theta}(z_u,u,c)$ , where  $z_{u}$  is the noisy version of the diffusion target  $z$  at diffusion timestep  $u$ , and  $c$  is the conditioning (note we denote diffusion timesteps as  $u$  to avoid conflict with  $t$ , which refers to video timesteps). For each diffusion timestep  $u$ , this model predicts the noise  $\epsilon$  added to  $z$ . This prediction is done in the latent space of the VQ encoder, and the diffusion model is implemented using a U-Net architecture [66] with interleaving attention layers [85] to incorporate conditioning on past frames. The use of attention enables learning of where to steal pixels from, eliminating the need for explicit tracking. Furthermore, this allows us to condition on previous frames while reusing the weights from diffusion models pre-trained on large-scale image datasets.
We work with  $256 \times 256$  images. The VQ encoder reduces the spatial dimension to  $64 \times 64$ , the resolution at which the denoising model is trained. The commonly used input for single-frame inpainting with a U-Net diffusion model is to simply concatenate the masked image, mask, and noisy target image along the channel dimension [63]. This gives an input shape of  $(2d + 1, 64, 64)$  where  $d$  is the dimension of the latent VQ codes. This U-Net model stacks blocks of convolutions, followed by self-attention within the spatial features. VIDM also concatenates the noisy target  $z_{u}$  with conditioning  $c$  (i.e. the context images,  $I_{t - h}, \ldots, I_{t}$ , and masks,  $m_{t - h}^{\mathrm{agent}}, \ldots, m_{t}^{\mathrm{agent}}$ ) however, we stack context frames (i.e.  $t - h$  through  $t - 1$ ) along a new dimension. As such, our U-Net accepts a  $(2d + 1, h + 1, 64, 64)$  tensor, where  $h$  is the number of additional context frames. We perform convolutions on the  $h + 1$  sets of spatial features in parallel but allow attention across all frames at attention blocks. Consequently, we can re-use the weights of an inpainting diffusion model trained on
Table 1: In-painting evaluation on held-out clips from Epic-Kitchens [12]. Use of strong generative priors and past frames allows our model to outperform past works that use only one or the other.
Inpainting Method	PSNR↑	SSIM↑	FID↓	Runtime ↓
Latent Diffusion [63]	28.29	0.931	27.29	12.5s / image
Latent Diffusion (fine-tuned)	28.27	0.931	27.50	12.5s / image
DLFormer [62]	26.98	0.922	51.74	106.4s / image
VIDM (Ours)	32.26	0.956	10.37	13.6s / image
VIDM trained with hands visible	31.81	0.953	11.04	13.6s / image
single images. The final spatial features are combined using a  $(h + 1)\times 1\times 1$  convolution to form the final spatial prediction. Following prior work, the loss is  $\mathbb{E}_{z,c,\epsilon \sim \mathcal{N}(0,1),u}\big[\| \epsilon -\epsilon_{\theta}(z_u,u,c)\| _1\big]$
Training Dataset. The data for training the model is extracted from Epic-Kitchens [12] and a subset of Ego4D [20] (kitchen videos). Crucially, we don't have real ground truth (i.e. videos with and without human hands) for training this model. We synthetically generate training data by masking out hand-shaped regions from videos and asking the model to inpaint these regions. We use hand segment sequences from VISOR [13] as the pool of hand-shaped masks. In addition, we also generate synthetic masks using the scheme from [63, 104]. We use a 3-frame history (i.e.  $h = 3$ ). Context images are drawn to be approximately 0.75s apart. We found it important to not have actual human hands as prediction targets for the diffusion model. Thus, we omit frames containing hands from Ego4D (using hand detector from [70]) and do not back-propagate loss on patches that overlap a hand in Epic-Kitchens (we use ground truth hand annotations on Epic-Kitchens frames from VISOR). We end up with 1.5 million training frames in total.
Model Training. We initialize our networks using pre-trained models. Specifically, we use the pretrained VQ encoder-decoder from [63] which is kept fixed. The latent diffusion model is pre-trained for single-frame inpainting on the Places [105] dataset and is finetuned for our multi-frame inpainting task. We train with a batch size of 48 for 600k iterations on 8 A40 GPUs for 12 days. At inference time we use 200 denoising steps to generate images.
Our overall model realizes the two desiderata in the design of a video inpainting model. First, conditioning on previous frames allows occluded regions to be filled using the appearance from earlier frames directly (if available). Second, the use of a strong data-driven generative prior (by virtue of starting from a diffusion model pre-trained on a large dataset) allows speculation of the content of the masked regions from the surrounding context.
We design experiments to test the inpainting abilities of VIDM (Sec. 5.1), and the utility of our factorized representation for different visual perception and robot learning tasks (Sec. 5.2 to Sec. 5.6). For the former, we assess the contribution of using a rich generative prior and conditioning on past frames. For the latter, we explore 5 benchmark tasks: object detection, 3D object reconstruction, affordance prediction, learning reward functions, and learning policies using learned rewards. We compare to alternate representations, specifically ones used in past papers for the respective tasks.
5.1 Reconstruction Quality Evaluation
We start by assessing the quality of our proposed inpainting model, VIDM. Following recent literature [36,41, 62], we evaluate the PSNR, SSIM and FID scores of inpainted frames against ground truth images.
Evaluation Dataset. We select 33 video clips that do not contain any hands from the 7 held-out participants from the Epic-Kitchens dataset [12]. These clips are on average 3.5 seconds long and span a total of 5811 frames. For each clip, we mask out a sequence of hand-shaped masks mined from the VISOR dataset [13]. The prediction problem for inpainting models is to reproduce the regions underneath the hand-shaped masks. This procedure simulates occlusion patterns that models will need to inpaint at test time, while also providing access to ground truth pixels for evaluation.
a) Original Image
b) LatentDiffusion FT [63]
c) DLFormer [62]
Figure 4: Our approach (VIDM) is able to correctly steal background information from past frames (top row, oranges on the bottom right) and also correctly reconstructs the wok handle using strong object appearance priors (bottom row).
d) VIDM (Ours)
Table 2: Average recall of detections from a COCO-trained Mask RCNN [24] on active objects (i.e. objects undergoing interaction) from VISOR [13]. See Sec. 5.2 for more details.
Image Used	ARall@1	ARall@5	ARall@10	AR0.5@1	AR0.5@5	AR0.5@10
Raw Image (i.e. It)	0.137	0.263	0.272	0.265	0.530	0.551
Masked Image (i.e. It with mtagent blacked out)	0.131	0.236	0.245	0.266	0.474	0.495
Ienv inpainted using Latent Diffusion [63]	0.149	0.259	0.270	0.299	0.517	0.541
Ienv inpainted using Latent Diffusion (finetuned)	0.154	0.262	0.271	0.305	0.519	0.540
Ienv inpainted using VIDM (Ours w/o factorization)	0.163	0.268	0.277	0.317	0.521	0.543
It and Ienv inpainted using VIDM (Ours w/ factorization)	0.170	0.379	0.411	0.334	0.681	0.735
Baselines. We compare against 3 baseline models: a) the single-frame latent diffusion inpainting model [63] trained on the Places [105] dataset that we initialize from, b) this model but finetuned on single images from the same Epic-Kitchens dataset that we use for our training, c) DLFormer [62], the current state-of-the-art video inpainting model. DLFormer trains one model per clip and sees every frame in the test clip, unlike our model which only sees the previous 3 frames for each prediction. Finally, we include one ablation: VIDM trained with hands visible in the training objective (i.e. gradients propagate to all pixels) instead of the objective used in VIDM (stop-gradient on pixels with hands, see Section 3 for details).
Results. Tab. 1 reports the results. We can see our model outperforms all baselines. Processing context frames with attention allows our model to improve upon single-frame diffusion baselines. Training with multiple frames and hands visible improves performance, but this training procedure requires the model to output hands during training. Consequently, inpainted regions sometimes include hands at test time (see Figure S5). Our model also outperforms DLFormer [62], while being  $8 \times$  faster and using only 3 frames of history. Fig. 4 shows example visualizations of inpainting results.
5.2 Application 1: Object Detection
The detection and recognition of objects being interacted with (by humans or robots) is made challenging due to occlusion caused by the hand / end-effector. While existing object detection datasets may include objects held and occluded by human hands, objects may be occluded in ways outside of the training distribution (e.g. when running COCO-trained detectors on egocentric videos). Furthermore, very few object detection datasets include examples of occlusion by a robot end-effector. We hypothesize that by removing the agent from the scene, off-the-shelf detectors may work better without requiring any additional training.
Protocol. We evaluate this hypothesis by testing COCO [39]-trained Mask RCNN detectors [24] on egocentric frames showing hand-object interaction from VISOR [13]. We only evaluate on object
Table 3: Average Precision on the Region-of-Interaction task and mAP for Grasps Afforded by Objects task from [19]. Using our model to remove the hands improves performance vs. placing a square mask over hands as done in [19]. See Sec. 5.4 for more details.
Image Used	Region of Interaction (RoI)	Grasps Afforded by Objects (GAO)
0% Slack	1% Slack
Masked image (It with mtagent blacked out) [19]	47.50 ± 2.77	54.03 ± 3.27	35.53 ± 3.67
Itenv (inpainted using VIDM) (Ours)	50.77 ± 0.81	57.10 ± 1.01	41.00 ± 3.99
classes that appear in both COCO and VISOR. VISOR [13] only annotates the active object and background objects have not been annotated. Thus, we measure performance using Average Recall (AR) as opposed to Average Precision. Specifically, we predict  $k$  ( $= 1, 5, 10$  in our experiments) boxes per class per image and report box-level Average Recall at 0.5 box overlap  $(\mathrm{AR}_{0.5})$  and integrated over box overlap thresholds  $\{0.5, 0.55, \dots, 0.95\} (\mathrm{AR}_{\mathrm{all}})$ .