SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Figure S2 we visualize how VIDM responds to corruptions in the input images to highlight VIDM's ability to copy information from the context images. For this visualization we use a sequence of images where the region occluded by the hand is visible in the context frames. We manually corrupt the context frames by masking out (with both black and red boxes) the region from which the model needs to draw information to inpaint properly. We see that when presented with corrupted context frames, VIDM copies the corrupted pixels (black or red) to represent the occluded region, while it properly inpaints the region when using the uncorrupted frames.
Qualitative visualizations in Figure S5 exhibit the failure mode of training with hands visible. Because this training procedure requires the model to output hands some of the time, it sometimes paints the hand like pixels back into the image.
Figure S3: An example batch for training VIDM. Columns 1-4: Input images to the network. Column 5: target image for reconstruction. Column 6: Masked regions on the target image. Column 7: Pixels with loss propagated (white pixels have loss, gray pixels have no loss). Note that hands that are masked in the target image (column 5) have no loss on them. See Section 4 for details.
c) DLFormer [62]
d)VIDM (Ours)
Figure S4: Additional visualizations of predictions from our method and baselines.
Training /w hands
VIDM
Latent Diffusion
Figure S5: Training while propagating loss from pixels containing the hand, the model learns to reconstruct hands in occluded regions (left). This behavior does not occur with our training procedure, which does not apply loss on pixels with hands (center), and does not appear in base single-image model that has no egocentric specific finetuning (right).