InstanceCap: Improving Text-to-Video Generation via Instance-aware
  Structured Caption
Authors: name: Tiehan Fan; name: Kepan Nan; name: Rui Xie; name: Penghao Zhou; name: Zhenheng Yang; name: Chaoyou Fu; name: Xiang Li; name: Jian Yang; name: Ying Tai
Published: 2024-12-12 13:48:40

Abstract: Text-to-video generation has evolved rapidly in recent years, delivering
remarkable results. Training typically relies on video-caption paired data,
which plays a crucial role in enhancing generation performance. However,
current video captions often suffer from insufficient details, hallucinations
and imprecise motion depiction, affecting the fidelity and consistency of
generated videos. In this work, we propose a novel instance-aware structured
caption framework, termed InstanceCap, to achieve instance-level and
fine-grained video caption for the first time. Based on this scheme, we design
an auxiliary models cluster to convert original video into instances to enhance
instance fidelity. Video instances are further used to refine dense prompts
into structured phrases, achieving concise yet precise descriptions.
Furthermore, a 22K InstanceVid dataset is curated for training, and an
enhancement pipeline that tailored to InstanceCap structure is proposed for
inference. Experimental results demonstrate that our proposed InstanceCap
significantly outperform previous models, ensuring high fidelity between
captions and videos while reducing hallucinations.