LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style
  Conditioned Image Generation
Authors: name: Donald Shenaj; name: Ondrej Bohdal; name: Mete Ozay; name: Pietro Zanuttigh; name: Umberto Michieli
Published: 2024-12-06 16:04:56

Abstract: Recent advancements in image generation models have enabled personalized
image creation with both user-defined subjects (content) and styles. Prior
works achieved personalization by merging corresponding low-rank adaptation
parameters (LoRAs) through optimization-based methods, which are
computationally demanding and unsuitable for real-time use on
resource-constrained devices like smartphones. To address this, we introduce
LoRA.rar, a method that not only improves image quality but also achieves a
remarkable speedup of over 4000times in the merging process. LoRA.rar
pre-trains a hypernetwork on a diverse set of content-style LoRA pairs,
learning an efficient merging strategy that generalizes to new, unseen
content-style pairs, enabling fast, high-quality personalization. Moreover, we
identify limitations in existing evaluation metrics for content-style quality
and propose a new protocol using multimodal large language models (MLLM) for
more accurate assessment. Our method significantly outperforms the current
state of the art in both content and style fidelity, as validated by MLLM
assessments and human evaluations.