BRAT: Bonus oRthogonAl Token for Architecture Agnostic Textual Inversion
Authors: name: James Baker
Published: 2024-08-08 23:04:26

Abstract: Textual Inversion remains a popular method for personalizing diffusion
models, in order to teach models new subjects and styles. We note that textual
inversion has been underexplored using alternatives to the UNet, and experiment
with textual inversion with a vision transformer. We also seek to optimize
textual inversion using a strategy that does not require explicit use of the
UNet and its idiosyncratic layers, so we add bonus tokens and enforce
orthogonality. We find the use of the bonus token improves adherence to the
source images and the use of the vision transformer improves adherence to the
prompt. Code is available at https://github.com/jamesBaker361/tex_inv_plus.