RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame
  Animated Sticker Generation
Authors: name: Zhiqiang Yuan, name: Ting Zhang, name: Ying Deng, name: Jiapei Zhang, name: Yeshuang Zhu, name: Zexi Jia, name: Jie Zhou, name: Jinchao Zhang

Recently, great progress has been made in video generation technology,
attracting the widespread attention of scholars. To apply this technology to
downstream applications under resource-constrained conditions, researchers
usually fine-tune the pre-trained models based on parameter-efficient tuning
methods such as Adapter or Lora. Although these methods can transfer the
knowledge from the source domain to the target domain, fewer training
parameters lead to poor fitting ability, and the knowledge from the source
domain may lead to the inference process deviating from the target domain. In
this paper, we argue that under constrained resources, training a smaller video
generation model from scratch using only million-level samples can outperform
parameter-efficient tuning on larger models in downstream applications: the
core lies in the effective utilization of data and curriculum strategy. Take
animated sticker generation (ASG) as a case study, we first construct a
discrete frame generation network for stickers with low frame rates, ensuring
that its parameters meet the requirements of model training under constrained
resources. In order to provide data support for models trained from scratch, we
come up with a dual-mask based data utilization strategy, which manages to
improve the availability and expand the diversity of limited data. To
facilitate convergence under dual-mask situation, we propose a
difficulty-adaptive curriculum learning method, which decomposes the sample
entropy into static and adaptive components so as to obtain samples from easy
to difficult. The experiment demonstrates that our resource-efficient dual-mask
training framework is quantitatively and qualitatively superior to
efficient-parameter tuning methods such as I2V-Adapter and SimDA, verifying the
feasibility of our method on downstream tasks under constrained resources. Code
will be available.