Hugging Face's logo

LGirrbach
/
ssr-merge-experiments

Text-to-Image
Diffusers
lora
dreambooth
sd3
flux
model-merging
Model card Files
xet
 Community

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

DreamBooth LoRAs for 30 Subjects on SD3.5-Medium and FLUX.1-dev

This repository holds 60 DreamBooth LoRA adapters. There is one adapter for each of the 30 subjects of the DreamBooth dataset (Ruiz et al., CVPR 2023) on each of two base models:

Folder Base model Adapters Adapted layers Size per adapter
sd35m/ stabilityai/stable-diffusion-3.5-medium 30 191 (all attention projections) 72 MB
flux/ black-forest-labs/FLUX.1-dev 30 342 (attention and MLP projections) 343 MB

All adapters use the same recipe and an identical, unambiguous trigger-phrase scheme. Each adapter was trained with its own random seed, so the adapters are fully independent, as community LoRAs would be. Together they form a 30-adapter pool. It includes several closely related subjects (seven dogs, two cats, three plushies, three shoes), which makes the pool suited to research on multi-LoRA merging, composition and interference.

Layout and usage 

sd35m/<subject>/pytorch_lora_weights.safetensors
flux/<subject>/pytorch_lora_weights.safetensors

The files use the standard diffusers LoRA format (keys transformer.<module>.lora_{A,B}.weight, fp32, rank 32, LoRA scaling 1, i.e. $\Delta W = BA$):

import torch
from diffusers import FluxPipeline  # or StableDiffusion3Pipeline with subfolder="sd35m/<subject>"

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16).to("cuda")
pipe.load_lora_weights("LGirrbach/ssr-merge-experiments", subfolder="flux/dog",
                       weight_name="pytorch_lora_weights.safetensors", adapter_name="dog")
image = pipe("a sks dog on the beach", num_inference_steps=28, guidance_scale=3.5,
             height=512, width=512).images[0]

Prompt every adapter with its trigger: a photo of sks <phrase> or a sks <phrase> ..., where <phrase> is given in the table below. The phrases are unique across the pool (e.g. sks chow chow, sks poodle, sks grey cat), so every adapter has its own trigger even when many adapters are loaded at once.

Training

The recipe follows the defaults of the official diffusers DreamBooth-LoRA scripts (train_dreambooth_lora_sd3.py / train_dreambooth_lora_flux.py, diffusers v0.41.0). The rank, learning rate and step count are the same as those of the FLUX.1-dev subject LoRAs in Wei et al., SSR-Merge: Subspace Signal Routing for Training-Free LoRA Merging in Diffusion Models.

Setting Value
Training data all reference images of the subject (4–6 per subject), 512 px, center crop
Instance prompt a photo of sks <phrase>
Rank / LoRA Ξ± 32 / 32 (scaling 1); Gaussian initialization of A
Target modules SD3.5-M: to_q/k/v, to_out.0, add_q/k/v_proj, to_add_out. FLUX: the same, plus ff.net.{0.proj,2} and ff_context.net.{0.proj,2}
Optimizer AdamW, lr 1e-4 constant, weight decay 1e-4, Ξ΅ 1e-8, gradient clipping 1.0
Steps / batch size 500 / 1
Noise schedule uniform timestep sampling (weighting_scheme=none)
Other no prior preservation, text encoders frozen. FLUX: guidance 1 during training, T5 length 512. SD3.5-M: T5 length 77
Seeds a different seed for each subject (independent initializations)

Evaluation

Each adapter is compared with the base model without an adapter. Both use the same prompts and seeds:

  • Prompts: a photo of sks <phrase>, a sks <phrase> on top of a wooden floor, a sks <phrase> on the beach, a sks <phrase> in the snow.
  • Images: 2 seeds per prompt, i.e. 8 images per subject. Each image is 512 Γ— 512 px with 28 sampling steps. Guidance is 5.0 (CFG) for SD3.5-M and 3.5 (distilled) for FLUX.1-dev.
  • DINO: cosine similarity between the DINOv2-base embedding of the generated image and the subject's reference images, averaged over all references (subject fidelity).
  • CLIP-I: the same with CLIP ViT-L/14 image embeddings.
  • CLIP-T: CLIP ViT-L/14 similarity between the image and the prompt with sks removed (prompt following).

Summary

Base model Subjects DINO base β†’ LoRA CLIP-I base β†’ LoRA CLIP-T base β†’ LoRA
SD3.5-Medium all 30 0.433 β†’ 0.671 0.702 β†’ 0.804 0.288 β†’ 0.285
SD3.5-Medium the 10 subjects of Wei et al. 0.374 β†’ 0.650 0.668 β†’ 0.798 0.283 β†’ 0.282
FLUX.1-dev all 30 0.470 β†’ 0.723 0.730 β†’ 0.833 0.282 β†’ 0.275
FLUX.1-dev the 10 subjects of Wei et al. 0.403 β†’ 0.711 0.698 β†’ 0.827 0.275 β†’ 0.266

Every adapter raises subject fidelity substantially over its base model. Prompt following changes very little: the mean CLIP-T falls by at most 0.009, so the adapters do not simply reproduce the backgrounds of their training images.

Comparison with Wei et al. (SSR-Merge). Wei et al. train FLUX.1-dev LoRAs for 10 of these subjects with the same rank, learning rate and step count. They report a single-LoRA "upper bound" of 0.744 DINO / 0.845 CLIP averaged over those subjects. Our FLUX.1-dev adapters reach 0.711 DINO / 0.827 CLIP-I on the same 10 subjects (0.723 / 0.833 on all 30). That is within 0.03 DINO and 0.02 CLIP of the published numbers.

The two evaluations are not identical, so the gap should not be read as a quality difference. Wei et al. do not fully specify their evaluation prompts, seeds, resolution or the DINOv2/CLIP variants. Our prompts also include three recontextualization prompts (wooden floor, beach, snow), which place the subject in new scenes and lower similarity to the reference images. Wei et al. report no SD3.5-Medium adapters. The SD3.5-Medium pool reaches slightly lower absolute similarity than the FLUX pool, but the gain over its base model is about as large (+0.24 vs. +0.25 DINO).

Per-subject results

† marks the 10 subjects used by Wei et al. "base" is the base model on the same prompts and seeds.

Subject (folder) Trigger phrase SD3.5-M DINO (base β†’ LoRA) SD3.5-M CLIP-I FLUX DINO (base β†’ LoRA) FLUX CLIP-I
bear_plushie † sks bear plushie 0.63 β†’ 0.68 0.63 β†’ 0.70 0.66 β†’ 0.73 0.64 β†’ 0.73
candle † sks candle 0.22 β†’ 0.54 0.61 β†’ 0.74 0.20 β†’ 0.69 0.62 β†’ 0.78
cat † sks cat 0.38 β†’ 0.65 0.80 β†’ 0.89 0.34 β†’ 0.77 0.83 β†’ 0.94
colorful_sneaker † sks colorful sneaker 0.57 β†’ 0.63 0.60 β†’ 0.71 0.58 β†’ 0.61 0.62 β†’ 0.77
dog † sks dog 0.08 β†’ 0.70 0.72 β†’ 0.88 0.20 β†’ 0.75 0.81 β†’ 0.91
poop_emoji † sks poop emoji 0.44 β†’ 0.69 0.70 β†’ 0.85 0.55 β†’ 0.72 0.77 β†’ 0.86
pink_sunglasses † sks pink sunglasses 0.61 β†’ 0.65 0.77 β†’ 0.84 0.46 β†’ 0.65 0.77 β†’ 0.84
red_cartoon † sks red cartoon 0.09 β†’ 0.74 0.56 β†’ 0.83 0.37 β†’ 0.73 0.59 β†’ 0.83
rc_car † sks rc car 0.21 β†’ 0.65 0.51 β†’ 0.70 0.16 β†’ 0.75 0.51 β†’ 0.75
teapot † sks teapot 0.53 β†’ 0.58 0.78 β†’ 0.83 0.51 β†’ 0.71 0.82 β†’ 0.86
backpack sks backpack 0.50 β†’ 0.60 0.75 β†’ 0.85 0.49 β†’ 0.61 0.75 β†’ 0.85
backpack_dog sks dog backpack 0.20 β†’ 0.61 0.60 β†’ 0.65 0.09 β†’ 0.74 0.66 β†’ 0.79
berry_bowl sks berry bowl 0.46 β†’ 0.81 0.76 β†’ 0.84 0.53 β†’ 0.83 0.80 β†’ 0.85
can sks beer can 0.66 β†’ 0.74 0.51 β†’ 0.58 0.60 β†’ 0.75 0.54 β†’ 0.59
cat2 sks grey cat 0.61 β†’ 0.84 0.82 β†’ 0.86 0.78 β†’ 0.86 0.84 β†’ 0.83
clock sks alarm clock 0.39 β†’ 0.58 0.77 β†’ 0.90 0.55 β†’ 0.65 0.90 β†’ 0.92
dog2 sks chow chow 0.61 β†’ 0.80 0.83 β†’ 0.89 0.70 β†’ 0.85 0.87 β†’ 0.91
dog3 sks poodle 0.38 β†’ 0.53 0.74 β†’ 0.79 0.22 β†’ 0.50 0.71 β†’ 0.81
dog5 sks dachshund 0.35 β†’ 0.72 0.76 β†’ 0.84 0.36 β†’ 0.80 0.78 β†’ 0.89
dog6 sks corgi puppy 0.66 β†’ 0.85 0.86 β†’ 0.89 0.66 β†’ 0.89 0.86 β†’ 0.91
dog7 sks retriever 0.29 β†’ 0.74 0.72 β†’ 0.87 0.35 β†’ 0.77 0.82 β†’ 0.88
dog8 sks border collie 0.47 β†’ 0.66 0.84 β†’ 0.87 0.53 β†’ 0.71 0.84 β†’ 0.87
duck_toy sks duck toy 0.55 β†’ 0.76 0.78 β†’ 0.85 0.63 β†’ 0.78 0.81 β†’ 0.84
fancy_boot sks fringe boot 0.50 β†’ 0.75 0.66 β†’ 0.76 0.68 β†’ 0.81 0.74 β†’ 0.85
grey_sloth_plushie sks sloth plushie 0.61 β†’ 0.64 0.72 β†’ 0.79 0.63 β†’ 0.74 0.71 β†’ 0.82
monster_toy sks monster toy 0.12 β†’ 0.55 0.53 β†’ 0.73 0.27 β†’ 0.61 0.54 β†’ 0.77
robot_toy sks robot toy 0.53 β†’ 0.71 0.70 β†’ 0.81 0.63 β†’ 0.76 0.73 β†’ 0.85
shiny_sneaker sks shiny sneaker 0.51 β†’ 0.67 0.63 β†’ 0.82 0.53 β†’ 0.67 0.62 β†’ 0.84
vase sks vase 0.27 β†’ 0.43 0.71 β†’ 0.80 0.28 β†’ 0.52 0.76 β†’ 0.82
wolf_plushie sks wolf plushie 0.58 β†’ 0.64 0.68 β†’ 0.75 0.57 β†’ 0.73 0.65 β†’ 0.81

Fidelity varies with the subject. On both models it is lowest for dog3 and vase (DINO 0.43–0.53); these are the only adapters below 0.55 on both models.

Licenses and intended use

The adapters are derivatives of their base models and inherit their licenses:

The training images come from the DreamBooth dataset, which documents the image sources and licenses. The adapters are released for research. Each one reproduces a specific object or pet from a handful of photos and is not intended for generating deceptive content.

Citation

If you use these adapters, please cite the DreamBooth dataset:

@inproceedings{ruiz2023dreambooth,
  title     = {DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation},
  author    = {Ruiz, Nataniel and Li, Yuanzhen and Jampani, Varun and Pritch, Yael and Rubinstein, Michael and Aberman, Kfir},
  booktitle = {CVPR},
  year      = {2023}
}
Downloads last month
-
Inference Providers NEW
Text-to-Image

Model tree for LGirrbach/ssr-merge-experiments

Base model

black-forest-labs/FLUX.1-dev
Adapter
(42,706)
this model