LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction
  Tuning
Authors: name: Bolin Lai; name: Xiaoliang Dai; name: Lawrence Chen; name: Guan Pang; name: James M. Rehg; name: Miao Liu
Published: 2023-12-06 19:02:40

Abstract: Generating instructional images of human daily actions from an egocentric
viewpoint serves a key step towards efficient skill transfer. In this paper, we
introduce a novel problem -- egocentric action frame generation. The goal is to
synthesize the action frame conditioning on the user prompt question and an
input egocentric image that captures user's environment. Notably, existing
egocentric datasets lack the detailed annotations that describe the execution
of actions. Additionally, the diffusion-based image manipulation models fail to
control the state change of an action within the corresponding egocentric image
pixel space. To this end, we finetune a visual large language model (VLLM) via
visual instruction tuning for curating the enriched action descriptions to
address our proposed problem. Moreover, we propose to Learn EGOcentric (LEGO)
action frame generation using image and text embeddings from VLLM as additional
conditioning. We validate our proposed model on two egocentric datasets --
Ego4D and Epic-Kitchens. Our experiments show prominent improvement over prior
image manipulation models in both quantitative and qualitative evaluation. We
also conduct detailed ablation studies and analysis to provide insights on our
method.