UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video
  Super-Resolution
Authors: name: Shian Du, name: Menghan Xia, name: Chang Liu, name: Quande Liu, name: Xintao Wang, name: Pengfei Wan, name: Xiangyang Ji
Organization: Kling Team

Cascaded video super-resolution has emerged as a promising technique for
decoupling the computational burden associated with generating high-resolution
videos using large foundation models. Existing studies, however, are largely
confined to text-to-video tasks and fail to leverage additional generative
conditions beyond text, which are crucial for ensuring fidelity in multi-modal
video generation. We address this limitation by presenting UniMMVSR, the first
unified generative video super-resolution framework to incorporate hybrid-modal
conditions, including text, images, and videos. We conduct a comprehensive
exploration of condition injection strategies, training schemes, and data
mixture techniques within a latent video diffusion model. A key challenge was
designing distinct data construction and condition utilization methods to
enable the model to precisely utilize all condition types, given their varied
correlations with the target video. Our experiments demonstrate that UniMMVSR
significantly outperforms existing methods, producing videos with superior
detail and a higher degree of conformity to multi-modal conditions. We also
validate the feasibility of combining UniMMVSR with a base model to achieve
multi-modal guided generation of 4K video, a feat previously unattainable with
existing techniques.