DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in
  Multimodal Cycles
Authors: name: Rui Zhao, name: Weijia Mao, name: Mike Zheng Shou

Adapting generative models to specific domains presents an effective solution
for satisfying specialized requirements. However, adapting to some complex
domains remains challenging, especially when these domains require substantial
paired data to capture the targeted distributions. Since unpaired data from a
single modality, such as vision or language, is more readily available, we
utilize the bidirectional mappings between vision and language learned by the
unified generative model to enable training on unpaired data for domain
adaptation. Specifically, we propose DoraCycle, which integrates two multimodal
cycles: text-to-image-to-text and image-to-text-to-image. The model is
optimized through cross-entropy loss computed at the cycle endpoints, where
both endpoints share the same modality. This facilitates self-evolution of the
model without reliance on annotated text-image pairs. Experimental results
demonstrate that for tasks independent of paired knowledge, such as
stylization, DoraCycle can effectively adapt the unified model using only
unpaired data. For tasks involving new paired knowledge, such as specific
identities, a combination of a small set of paired image-text examples and
larger-scale unpaired data is sufficient for effective domain-oriented
adaptation. The code will be released at https://github.com/showlab/DoraCycle.