4Fever4/siwa-kershef-sdxl-lora

Siwa Kershef Architecture — SDXL LoRA (style-isolation experiment)

A LoRA for SDXL 1.0 that teaches the earthen kershef architecture of Siwa Oasis, Egypt, behind the trigger token k3rshef, evaluated specifically for style bleeding into other architectural styles.

Dataset (112 curated, hand-captioned images, per-image licence): 4Fever4/siwa-kershef-architecture

Usage

import torch
from diffusers import StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to("cuda")
pipe.load_lora_weights("4Fever4/siwa-kershef-sdxl-lora", weight_name="siwa_kershef_sdxl_lora_v1.safetensors", adapter_name="siwa")
pipe.set_adapters(["siwa"], adapter_weights=[0.6])   # recommended strength, see evaluation
img = pipe("k3rshef, exterior of a two-storey house, projecting palm-trunk beams, small window openings, restored, photo",
           num_inference_steps=30, guidance_scale=6.5).images[0]

Prompt vocabulary the LoRA was captioned with (usable at inference): palm-trunk beams, thick tapered columns, small window openings, external staircase, crenellated parapet, tapered minaret, covered passage; condition: restored, partially restored, newly built, eroded ruin.

Training

Base	SDXL 1.0 base, fp16-fix VAE
Network	LoRA, UNet only, rank 16, alpha 8
Data	112 images, aspect-ratio bucketing 640–1536 px, 1024² base resolution
Optimiser	AdamW8bit, lr 1e-4, cosine, 100 warm-up steps, min-SNR γ=5, noise offset 0.0357
Steps	v1: 1,600 (≈14 epochs); v2: 2,000 with 48 regularisation images; batch 1, checkpoints every 400
Hardware	single RTX 4060 8 GB (bf16, gradient checkpointing, cached latents + text-encoder outputs), ≈38 min

Exact command: train.sh, dataset config: dataset.toml (kohya-ss/sd-scripts).

Evaluation — does the style stay behind its trigger?

scripts/evaluate.py generates the same prompts with the same seeds with base SDXL and with the LoRA loaded.

• Target prompts (with k3rshef) — the LoRA should pull these toward Siwa.
• Control prompts (no trigger, LoRA still loaded): Al-Qasr Dakhla (a neighbouring earthen style — the hardest case), Nubian Aswan, modern New Cairo, Mamluk Cairo, Scandinavian — the LoRA should not change these.

Metrics (CLIP ViT-L/14, cosine):

• siwa_shift = similarity to the training-set centroid (LoRA) − same (base). Target ↑, control ≈ 0.
• base_vs_lora_sim = similarity between the base image and the LoRA image for the same prompt+seed. Control ≈ 1 means untouched.

Summary

run	target siwa_shift ↑	control siwa_shift → 0	control base-vs-LoRA sim → 1
v1 @1.0	+0.058	+0.044	0.891
v1 @0.6 (recommended)	+0.049	+0.017	0.945
v2 @1.0	+0.007	+0.009	0.889
v2 @1.5	-0.006	-0.012	0.838

Per prompt (siwa_shift / base-vs-LoRA similarity)

prompt	v1 @1.0	v1 @0.6 (recommended)	v2 @1.0	v2 @1.5
T1_house	+0.071 / 0.810	+0.061 / 0.835	+0.019 / 0.870	-0.003 / 0.847
T2_alley	+0.008 / 0.849	+0.039 / 0.915	-0.022 / 0.888	-0.044 / 0.852
T3_interior	+0.067 / 0.785	+0.047 / 0.865	+0.023 / 0.872	+0.027 / 0.820
T4_mosque	+0.065 / 0.800	+0.077 / 0.879	+0.023 / 0.862	+0.010 / 0.841
T5_hotel	+0.077 / 0.855	+0.018 / 0.910	-0.008 / 0.860	-0.018 / 0.824
C1_alqasr	+0.029 / 0.961	+0.013 / 0.969	-0.004 / 0.928	-0.021 / 0.890
C2_nubian	+0.034 / 0.903	+0.013 / 0.932	-0.016 / 0.893	-0.041 / 0.826
C3_modern	+0.097 / 0.813	+0.056 / 0.897	+0.054 / 0.892	+0.045 / 0.823
C4_mamluk	+0.048 / 0.877	+0.005 / 0.964	+0.000 / 0.820	-0.035 / 0.774
C5_nordic	+0.010 / 0.899	-0.003 / 0.964	+0.008 / 0.914	-0.008 / 0.878

Rows: base SDXL / v1 @1.0 / v1 @0.6. Columns: house + mosque (target, with trigger), Nubian, modern New Cairo, Mamluk (controls, no trigger). Same seed.

What happened — three iterations

1. v1 @1.0 learned the style strongly (target +0.058) but bled into neighbouring Egyptian prompts (control +0.044): modern New Cairo turned into a beige earthen block, Nubian lost its blue paint and composition, Mamluk domes and carving flattened into mud walls. Scandinavian and — notably — Al-Qasr (0.96 similarity) stayed stable, so the leak is semantic ("Egypt / desert → kershef"), not global.
2. v2 (prior-preservation): retrained with 48 base-SDXL regularisation images of neighbouring styles (Nubian, modern Cairo, Mamluk, Ottoman, Mediterranean, riad, desert resort …, captioned without the trigger, wording/seeds disjoint from the eval prompts). Bleeding dropped to +0.009, but the target style collapsed too (+0.007) — the regulariser was too strong for a 112-image set at 2,000 steps. Raising v2 to 1.5 made things worse. Negative result, kept for transparency.
3. v1 @0.6 — recommended. Keeps 84 % of the target shift (+0.049) while cutting control bleeding by 61 % (+0.044 → +0.017); controls stay 0.945 similar to base (Mamluk 0.964, Nordic 0.964, Al-Qasr 0.969). Remaining weak spot: modern New Cairo still drifts (+0.056).

What I would do next

• Weaker prior preservation (fewer reg repeats / prior_loss_weight 0.3–0.5) instead of 1:1, then re-sweep scale.
• Add negative-pair captions for the modern case (e.g. modern glass building reg images specifically).
• More intact-building data (the dataset is still 45 % ruins) and a human rating pass alongside CLIP.
• Port the same data / captions / evaluation to a Flux LoRA on a 24 GB GPU.

Limitations

• One-day experiment; 112 images, 3 seeds per prompt — the numbers are indicative, not a benchmark.
• SDXL, not Flux: Flux LoRA training does not fit an 8 GB GPU. The same data/captions/evaluation pipeline transfers directly to Flux (e.g. ai-toolkit) on a 24 GB GPU.
• CLIP similarity is a proxy; the grids in eval/ should be looked at, not only the numbers.