Qwen Image 2.1 LoRA experiments

Built with Qwen. A curated collection of SimpleTuner training experiments: what ordinary Domokun training changes, what an assistant LoRA changes, and what happens when a separate concept LoRA is trained alongside a frozen assistant. Selected weights, original validation images, independent inference comparisons, and configuration recipes are included.

These are experimental adapters and qualitative observations, not a leaderboard or a recommended finished Domokun release. We include failed examples because subject preservation, concept learning, and attractive individual samples are different outcomes.

Latest: multi-scale training and REPA

Our strongest qualitative result in this series is the 4,000-update multi-scale REPA checkpoint with assistant v2 and synthetic regularisation. The inspected beach and bicycle samples express Domokun with coherent structure and fine texture at 1024px, and the beach sample also works at 512px. The unrelated portrait remains coherent. The comparisons below retain all five prompts, including the fox, so concept acquisition can be judged alongside image quality and spillover.

This completed REPA checkpoint used fixed flow shift 3. Auto shift was tested in a separate multi-scale run without REPA. A fresh 4,000-update run combining REPA and auto shift is underway; its results are not included here. The newer SimpleTuner examples combine those settings, but should not be mistaken for the recorded recipe of an earlier checkpoint.

Multi-scale + REPA, fixed shift, step 4000, character_beach, 1024px output; base left, trained right

All five runs below use frozen assistant v2, rank/alpha 32, BF16, batch 1, AdamW BF16 at 1e-4, 25 warmup updates, norm clipping at 1.0, and interval-2 gradient checkpointing. Regularisation targets the bare base prediction, with both concept and assistant adapters disabled for the target forward. The assistant remains frozen for the student forward and is disabled for validation. Audits checked these states and the parent prediction against an independent base forward. Earlier assistant-plus-regularisation rows on this page used an assisted parent target; they are a different experiment.

Variant Training resolution REPA Flow shift Selected LoRA checkpoints
512px control 512 No 3 2,000
512px + REPA 512 Yes 3 1,000, 2,000
Multi-scale, fixed shift 512 + 1024 No 3 2,000, 4,000
Multi-scale, auto shift 512 + 1024 No Auto 2,000, 4,000
Multi-scale + REPA, fixed shift 512 + 1024 Yes 3 2,000, 4,000

Multi-scale training uses the ordinary probabilistic sampler: Domokun-512, Domokun-1024, synthetic-512 and synthetic-1024 each have probability 0.25. The three completed multi-scale runs share the same initial concept weights and actual batch counts: 1,005 Domokun-512, 1,014 Domokun-1024, 1,036 synthetic-512 and 945 synthetic-1024. Four thousand updates preserve approximately the same exposure per resolution as the 2,000-update single-resolution runs. These experiments used square training buckets; the newer aspect-preserving example datasets are a separate recipe update.

REPA uses frozen DINOv2-G patch features at encoder size 518, transformer block 8, weight 0.5 and spatial alignment, with temporal distance 0. The trainable projector and nonzero gradients into the captured hidden states and concept LoRA were checked. In the auto-shift control, Qwen 2.1's one-latent-pixel patch size gives effective shifts of approximately 1.7138 at 512px and 2.0008 at 1024px. No custom alternating sampler was used.

Every grid uses CPU-generator seed 42, 40 inference steps and CFG 1, with the original pinned Qwen VAE and full-frame decoding. Base is on the left; the concept adapter, with the assistant disabled, is on the right. These samples do not use the newer texture-fixed decoder. They are saved training validations, not newly generated independent-seed tests. Each variant has one training seed; the qualitative preference above is not a quantitative ranking.

Final checkpoints: matched controls

Expand: 512px training, without versus with REPA, 512px output
Prompt512px control, 2,000512px + REPA, 2,000
character beach512px control, step 2000, character_beach, 512px output; base left, trained right512px + REPA, step 2000, character_beach, 512px output; base left, trained right
character scarf512px control, step 2000, character_scarf, 512px output; base left, trained right512px + REPA, step 2000, character_scarf, 512px output; base left, trained right
character bicycle512px control, step 2000, character_bicycle, 512px output; base left, trained right512px + REPA, step 2000, character_bicycle, 512px output; base left, trained right
prior fox512px control, step 2000, prior_fox, 512px output; base left, trained right512px + REPA, step 2000, prior_fox, 512px output; base left, trained right
prior portrait512px control, step 2000, prior_portrait, 512px output; base left, trained right512px + REPA, step 2000, prior_portrait, 512px output; base left, trained right
Expand: 512px training, without versus with REPA, 1024px output
Prompt512px control, 2,000512px + REPA, 2,000
character beach512px control, step 2000, character_beach, 1024px output; base left, trained right512px + REPA, step 2000, character_beach, 1024px output; base left, trained right
character scarf512px control, step 2000, character_scarf, 1024px output; base left, trained right512px + REPA, step 2000, character_scarf, 1024px output; base left, trained right
character bicycle512px control, step 2000, character_bicycle, 1024px output; base left, trained right512px + REPA, step 2000, character_bicycle, 1024px output; base left, trained right
prior fox512px control, step 2000, prior_fox, 1024px output; base left, trained right512px + REPA, step 2000, prior_fox, 1024px output; base left, trained right
prior portrait512px control, step 2000, prior_portrait, 1024px output; base left, trained right512px + REPA, step 2000, prior_portrait, 1024px output; base left, trained right
Expand: multi-scale fixed shift versus auto shift versus REPA, 512px output
PromptMulti-scale, fixed shift, 4,000Multi-scale, auto shift, 4,000Multi-scale + REPA, fixed shift, 4,000
character beachMulti-scale, fixed shift, step 4000, character_beach, 512px output; base left, trained rightMulti-scale, auto shift, step 4000, character_beach, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_beach, 512px output; base left, trained right
character scarfMulti-scale, fixed shift, step 4000, character_scarf, 512px output; base left, trained rightMulti-scale, auto shift, step 4000, character_scarf, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_scarf, 512px output; base left, trained right
character bicycleMulti-scale, fixed shift, step 4000, character_bicycle, 512px output; base left, trained rightMulti-scale, auto shift, step 4000, character_bicycle, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_bicycle, 512px output; base left, trained right
prior foxMulti-scale, fixed shift, step 4000, prior_fox, 512px output; base left, trained rightMulti-scale, auto shift, step 4000, prior_fox, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, prior_fox, 512px output; base left, trained right
prior portraitMulti-scale, fixed shift, step 4000, prior_portrait, 512px output; base left, trained rightMulti-scale, auto shift, step 4000, prior_portrait, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, prior_portrait, 512px output; base left, trained right
Expand: multi-scale fixed shift versus auto shift versus REPA, 1024px output
PromptMulti-scale, fixed shift, 4,000Multi-scale, auto shift, 4,000Multi-scale + REPA, fixed shift, 4,000
character beachMulti-scale, fixed shift, step 4000, character_beach, 1024px output; base left, trained rightMulti-scale, auto shift, step 4000, character_beach, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_beach, 1024px output; base left, trained right
character scarfMulti-scale, fixed shift, step 4000, character_scarf, 1024px output; base left, trained rightMulti-scale, auto shift, step 4000, character_scarf, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_scarf, 1024px output; base left, trained right
character bicycleMulti-scale, fixed shift, step 4000, character_bicycle, 1024px output; base left, trained rightMulti-scale, auto shift, step 4000, character_bicycle, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_bicycle, 1024px output; base left, trained right
prior foxMulti-scale, fixed shift, step 4000, prior_fox, 1024px output; base left, trained rightMulti-scale, auto shift, step 4000, prior_fox, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, prior_fox, 1024px output; base left, trained right
prior portraitMulti-scale, fixed shift, step 4000, prior_portrait, 1024px output; base left, trained rightMulti-scale, auto shift, step 4000, prior_portrait, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, prior_portrait, 1024px output; base left, trained right

Learning progress with REPA

Expand: 512px + REPA, learning progress, 512px output
PromptStep 1,000Step 2,000
character beach512px + REPA, step 1000, character_beach, 512px output; base left, trained right512px + REPA, step 2000, character_beach, 512px output; base left, trained right
character scarf512px + REPA, step 1000, character_scarf, 512px output; base left, trained right512px + REPA, step 2000, character_scarf, 512px output; base left, trained right
character bicycle512px + REPA, step 1000, character_bicycle, 512px output; base left, trained right512px + REPA, step 2000, character_bicycle, 512px output; base left, trained right
prior fox512px + REPA, step 1000, prior_fox, 512px output; base left, trained right512px + REPA, step 2000, prior_fox, 512px output; base left, trained right
prior portrait512px + REPA, step 1000, prior_portrait, 512px output; base left, trained right512px + REPA, step 2000, prior_portrait, 512px output; base left, trained right
Expand: 512px + REPA, learning progress, 1024px output
PromptStep 1,000Step 2,000
character beach512px + REPA, step 1000, character_beach, 1024px output; base left, trained right512px + REPA, step 2000, character_beach, 1024px output; base left, trained right
character scarf512px + REPA, step 1000, character_scarf, 1024px output; base left, trained right512px + REPA, step 2000, character_scarf, 1024px output; base left, trained right
character bicycle512px + REPA, step 1000, character_bicycle, 1024px output; base left, trained right512px + REPA, step 2000, character_bicycle, 1024px output; base left, trained right
prior fox512px + REPA, step 1000, prior_fox, 1024px output; base left, trained right512px + REPA, step 2000, prior_fox, 1024px output; base left, trained right
prior portrait512px + REPA, step 1000, prior_portrait, 1024px output; base left, trained right512px + REPA, step 2000, prior_portrait, 1024px output; base left, trained right
Expand: Multi-scale + REPA, fixed shift, learning progress, 512px output
PromptStep 2,000Step 4,000
character beachMulti-scale + REPA, fixed shift, step 2000, character_beach, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_beach, 512px output; base left, trained right
character scarfMulti-scale + REPA, fixed shift, step 2000, character_scarf, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_scarf, 512px output; base left, trained right
character bicycleMulti-scale + REPA, fixed shift, step 2000, character_bicycle, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_bicycle, 512px output; base left, trained right
prior foxMulti-scale + REPA, fixed shift, step 2000, prior_fox, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, prior_fox, 512px output; base left, trained right
prior portraitMulti-scale + REPA, fixed shift, step 2000, prior_portrait, 512px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, prior_portrait, 512px output; base left, trained right
Expand: Multi-scale + REPA, fixed shift, learning progress, 1024px output
PromptStep 2,000Step 4,000
character beachMulti-scale + REPA, fixed shift, step 2000, character_beach, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_beach, 1024px output; base left, trained right
character scarfMulti-scale + REPA, fixed shift, step 2000, character_scarf, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_scarf, 1024px output; base left, trained right
character bicycleMulti-scale + REPA, fixed shift, step 2000, character_bicycle, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, character_bicycle, 1024px output; base left, trained right
prior foxMulti-scale + REPA, fixed shift, step 2000, prior_fox, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, prior_fox, 1024px output; base left, trained right
prior portraitMulti-scale + REPA, fixed shift, step 2000, prior_portrait, 1024px output; base left, trained rightMulti-scale + REPA, fixed shift, step 4000, prior_portrait, 1024px output; base left, trained right

The release adds nine concept checkpoints and 90 original grids. The intermediate no-REPA multi-scale and auto-shift samples are also available in their asset folders. Recipes explicitly record the original VAE and resolved flow-shift settings; current SimpleTuner includes the DINO patch-feature and Qwen auto-shift corrections used by these experiments.

For the REPA runs, the four training-only projector tensors are stored separately as repa_projector.safetensors beside the inference adapter. The pytorch_lora_weights.safetensors file contains the 256 LoRA tensors only, each verified unchanged from the saved training checkpoint. The original checkpoint hash and exported artifact hashes are recorded in artifacts.json. The projector is not needed for inference, and these files are not complete optimizer/data-loader resume checkpoints.

Extended training degradation test

The Qwen Image 2.1 photo-aesthetics 50k experiment extends the assistant-v2 tests to 50,000 photo-training updates at 512px, using photo-aesthetics as a downstream dataset that was not configured during assistant training. Together with the earlier controls, its preserved coherence and quality demonstrate assistant v2's protection through prolonged training in this setup, with the assistant disabled at inference.

The dedicated repository includes 10k, 20k, 30k, 40k and 50k checkpoints, all six validation prompts rendered at both 512px and 1024px against the base model, and a separate 1024px-trained reference at every 1k checkpoint through 10k. It includes recipes, provenance and expandable comparisons; dataset-level separation is documented, while image-level overlap with assistant sources has not been audited.

What to take from these experiments

  • A model can learn Domokun without reliably expressing it under every prompt. Adding “character” after 🟫, together with true two-pass CFG 4, materially changed the earlier assisted run's results. Changing both at once does not isolate either effect.
  • Assess assistant runs primarily on image coherence and quality. Track concept acquisition and leakage separately: a toothy fox can reveal concept spillover without establishing a failure of the assistant’s intended role.
  • Damage with an assistant applied is observable; beneficial downstream absorption of that damage is a hypothesis. The assistant's own images cannot establish whether it is useful.
  • Synthetic regularisation is a useful alternative worth investigating. It retained conventional fox mouths in the two held-out images while learning Domokun-like figures, but colour and shape errors remained.
  • With assistant v2, both 512px and 1024px training learned the character; 512px retained brown colouring better in the inspected beach/bicycle scenes. Both still showed CFG-1 fox tooth leakage.
  • A higher learning rate was not a reliable shortcut: the documented 3e-4 assisted run severely degraded by update 500.

How the assistant is used

There are two distinct stages. First, train the assistant adapter against ordinary image targets. In the second stage, freeze that adapter at strength 1.0, add a fresh trainable concept adapter, and train the concept adapter on Domokun alone. Disable the assistant for validation and final concept inference; keep the learned concept adapter.

The working idea is to put changes associated with losing the base model's distilled behaviour into a removable adapter. This is not an explicit loss that separates “damage” from “concept,” and we have not measured a general preservation guarantee. Assistant-only images below show the base with the assistant applied. Downstream images show the base with only the learned concept adapter. There is no negative assistant-strength subtraction at inference.

Experiments selected

All concept runs use the 27-image Domokun dataset and the literal caption 🟫, rank/alpha 32/32, batch/accumulation 1/1, training seed 42, BF16, corrected adamw_bf16, norm clipping 1.0, and a constant learning rate after 25 warmup updates, unless a row says otherwise. The frozen base is pinned to 790c92633540aa0cb11d9abf19eb46d861714758. Full-frame VAE encoding/decoding is used.

Experiment Training target/data Resolution Updates LR Assistant during training
Unassisted control Domokun only 512 500 and 1,000 shown 1e-4 None
Assistant v1 Base-generated images, CC12M-derived captions 512–2048, multiple aspect ratios 1,000 1e-4 This is the adapter being trained
Assistant v2 Synthetic + real CC12M + real e621 512–2048, multiple aspect ratios 1,000 1e-4 This is the adapter being trained
Assisted v1 concept Domokun only 512 1,000 1e-4 Frozen v1
Assisted v2 concept Domokun only 512 1,000 1e-4 Frozen v2
Assisted v2 concept Domokun only 1024 1,000 1e-4 Frozen v2
Synthetic regularisation Domokun + base-prediction matching on synthetic images 512 2,000 total: 1,012 concept + 988 regularisation 1e-4 None
Assisted + regularisation Domokun + assistant-parent matching on synthetic images 512 2,000 total: 904 concept + 1,096 regularisation 1e-4 Frozen v2
Assisted + regularisation Domokun + assistant-parent matching on synthetic images 1024 2,000 total: 904 concept + 1,096 regularisation 1e-4 Frozen v2
Higher-LR failure Domokun only 512 500 shown 3e-4 Frozen v1

The two new v2 concept runs share their configuration except training resolution and output identifiers, and start from identical concept-adapter weights. They do not include CC12M, e621, or synthetic regularisation. Resizing this small dataset to 1024 does not add source detail. Both are evaluated at 512 so output resolution does not become another variable.

1. Baseline training without an assistant

The unassisted 512px run produced a recognisable Domokun in the beach sample at update 500, with unusual geometry. The scarf sample still involved a person. By update 1,000, both original scene prompts returned people, while an unrelated fox acquired a large toothed mouth. This illustrates why a promising intermediate sample is not enough to declare training successful.

The original grids below have the base on the left and trained checkpoint on the right. They use CPU-generator seed 42, 40 steps, CFG 1, and 512px panels. The grid font may render the emoji as a missing-glyph box; the actual prompt contains 🟫.

Expand: unassisted training at 500 and 1,000 updates
Prompt / settingUpdate 500Update 1,000
domo_beachdomo_beachdomo_beach
domo_scarfdomo_scarfdomo_scarf
prior_foxprior_foxprior_fox
prior_portraitprior_portraitprior_portrait

The independent re-render below uses GPU-generator seed 123, 40 steps, and CFG 4. It includes both original prompts and the explicit character noun. These are a separate inference protocol from the periodic grids. With the character noun and CFG 4, even this unassisted final checkpoint produces Domokun-like imagery: the reviewed beach sample has distorted facial placement and an extra head attachment; the scarf sample is bear-like. The fox is conventional at this setting. Thus failure under the original prompts must not be presented as proof that the unassisted run learned nothing.

Expand: base versus unassisted final checkpoint, CFG 4
Prompt / settingBaseUnassisted 512, update 1,000
domo_beachdomo_beachdomo_beach
character_beachcharacter_beachcharacter_beach
character_scarfcharacter_scarfcharacter_scarf
prior_foxprior_foxprior_fox
prior_portraitprior_portraitprior_portrait

2. What training the assistant does to the base

v1 used targets generated by the base model itself: a reusable 10,000-image pool, with CC12M long_caption prompts, 40 generation steps and CFG 1. Its 1,000 training updates sample that pool rather than consuming all 10,000 images. It used no real CC12M/e621 image targets and no Domokun backend.

v2 tested whether mixing in real images produces a more useful training assistant. Its actual 1,000 batches were 338 synthetic, 331 real CC12M, and 331 real e621. The synthetic backends were capped at 128 samples per aspect bucket; each real-data resolution backend was capped at 256 samples. These are bounded pilot subsets, not passes over either full real dataset, and images can be reused across resolution backends. Real CC12M captions use long_caption. No Domokun backend was configured; a caption check rejected Domokun-name/trigger matches, but this does not establish that no unlabelled character appears in real images.

With v2 applied, the inspected CFG-1 fox at seed 123 has strongly distorted anatomy. CFG 4 produces a coherent fox for that seed. Portraits remain coherent despite appearance changes. This is evidence of a changed model response, not proof that v2 will protect a downstream concept adapter. Under the same CFG-1 fox prompt and seed, v1 remains a coherent fox while v2 is strongly malformed; this is a specific observed difference, not a general degradation score. The common-setting comparison below also prevents confusing v1's original 1024px validation with v2's 512px validation.

Expand: base, assistant v1 and assistant v2 at identical inference settings
Prompt / settingBaseBase + v1Base + v2
prior_fox, CFG 1, seed 123prior_fox, CFG 1, seed 123prior_fox, CFG 1, seed 123prior_fox, CFG 1, seed 123
prior_fox, CFG 4, seed 123prior_fox, CFG 4, seed 123prior_fox, CFG 4, seed 123prior_fox, CFG 4, seed 123
prior_portrait, CFG 1, seed 123prior_portrait, CFG 1, seed 123prior_portrait, CFG 1, seed 123prior_portrait, CFG 1, seed 123
prior_portrait, CFG 4, seed 123prior_portrait, CFG 4, seed 123prior_portrait, CFG 4, seed 123prior_portrait, CFG 4, seed 123
domo_beach, CFG 1, seed 123domo_beach, CFG 1, seed 123domo_beach, CFG 1, seed 123domo_beach, CFG 1, seed 123
domo_beach, CFG 4, seed 123domo_beach, CFG 4, seed 123domo_beach, CFG 4, seed 123domo_beach, CFG 4, seed 123
Expand: original assistant training grids — different output resolutions
Prompt / settingv1: 1024px panelsv2: 512px panels
foxfoxfox
portraitportraitportrait

3. Teaching Domokun alongside a frozen assistant

The earlier v1-assisted 512px run learned the appearance, but original “A photo of 🟫 ...” scene prompts often returned people or people holding a character. Using “A photo of 🟫 character ...” with two-pass CFG 4 produced recognisable Domokun across four contexts and two held-out seeds. Imperfect body shapes, extra attachments, and bicycle errors remained; one fox seed developed exaggerated teeth.

The following held-out screen compares that checkpoint with the untrained base, not with an unassisted concept adapter. It demonstrates concept acquisition but does not by itself prove the assistant caused an improvement.

Expand: base versus v1-assisted concept, two held-out seeds, CFG 4
Prompt / settingBaseConcept trained with v1; v1 disabled
character_beach, seed 123character_beach, seed 123character_beach, seed 123
character_beach, seed 777character_beach, seed 777character_beach, seed 777
character_scarf, seed 123character_scarf, seed 123character_scarf, seed 123
character_scarf, seed 777character_scarf, seed 777character_scarf, seed 777
character_bicycle, seed 123character_bicycle, seed 123character_bicycle, seed 123
character_bicycle, seed 777character_bicycle, seed 777character_bicycle, seed 777
character_umbrella, seed 123character_umbrella, seed 123character_umbrella, seed 123
character_umbrella, seed 777character_umbrella, seed 777character_umbrella, seed 777
prior_fox, seed 123prior_fox, seed 123prior_fox, seed 123
prior_fox, seed 777prior_fox, seed 777prior_fox, seed 777

Assistant v2: 512 versus 1024 training

Both paired runs completed all 1,000 updates, with identical initial concept-adapter hashes and verified frozen assistant weights. The assistant is disabled for every image below. At CFG 4, seed 123, both checkpoints produce Domokun-like figures in the beach, scarf, bicycle and umbrella scenes. The 512px run retains brown colouring on the beach and bicycle, while the 1024px run is burgundy/purple in all four. Both have burgundy scarf/umbrella samples and imperfect shapes. The beach colour difference also appears at seed 42.

The periodic CFG-1 fox changes from a conventional mouth at update 500 to obvious triangular teeth at update 1,000 in both runs. Their CFG-4 fox samples at seeds 42 and 123 are more conventional. Thus v2 does not eliminate concept spillover, and attractive CFG-4 examples should not conceal the CFG-1 failure. This limited comparison favours 512px for colour fidelity in the inspected scenes; it does not establish a universal resolution recommendation or isolate the assistant's effect.

Expand: downstream v2 comparison at common 512px inference, CFG 4, seed 123
Prompt / setting512px training1024px training
exact_triggerexact_triggerexact_trigger
character_beachcharacter_beachcharacter_beach
character_scarfcharacter_scarfcharacter_scarf
character_bicyclecharacter_bicyclecharacter_bicycle
character_umbrellacharacter_umbrellacharacter_umbrella
prior_foxprior_foxprior_fox
prior_portraitprior_portraitprior_portrait
Expand: v2 periodic history — concept learning and fox leakage at CFG 1
Prompt / setting512px, update 500512px, update 10001024px, update 5001024px, update 1000
character_beachcharacter_beachcharacter_beachcharacter_beachcharacter_beach
character_scarfcharacter_scarfcharacter_scarfcharacter_scarfcharacter_scarf
prior_foxprior_foxprior_foxprior_foxprior_fox
prior_portraitprior_portraitprior_portraitprior_portraitprior_portrait

4. An alternative: synthetic regularisation without an assistant

This run mixes Domokun (is_regularisation_data=false) with 834 synthetic images from the generated dataset's 512×512 bucket (is_regularisation_data=true), using equal backend sampling weights. Under this setting, SimpleTuner creates detached targets from the frozen base with the trainable adapter disabled, at the same noisy inputs. It is base-prediction matching, not simply ordinary denoising on another image dataset.

It finished with 1,012 Domokun updates and 988 regularisation updates. Repeats keep both backends eligible during sampling; the configured ratio is not equivalent to matching epochs. Compared with the 1,000-update assisted runs, concept exposure is approximately matched, but total optimizer updates and preservation mechanism differ.

With the character noun and CFG 4, the held-out images show Domokun-like figures with burgundy/purple colour drift and shape errors. One bicycle image becomes a person with a Domokun backpack. Both held-out fox mouths remain conventional. This is a promising preservation signal in a small sample, not proof of broad drift prevention.

The three-way table also includes the v2-assisted 512px concept checkpoint at update 1,000, with the assistant disabled. Its seed-123 images come from the existing independent evaluation; five additional seed-777 images complete the same prompt/seed screen. Every new image used 512px output, 40 steps, CUDA-generator seed 777, full-frame VAE decoding and an audited 80 transformer forwards for true CFG 4. The generation receipt pins the base revision and concept-checkpoint hash; the render script reproduces the additional images.

At both seeds, v2 produces brown beach/bicycle characters but burgundy/purple scarf/umbrella characters. Its seed-777 fox has conspicuous triangular teeth, while its seed-123 fox has a smaller smile. Both v1 and v2 therefore show a toothy fox in this screen; the synthetic-regularisation foxes have conventional muzzles. Concept leakage is a separate observation from coherence and perceptual quality. These few images do not establish an overall quality ranking or a causal advantage: the runs differ in preservation method and total optimizer updates, and each has only one training seed.

Expand: assisted v1, assisted v2 and synthetic regularisation, held-out CFG 4
Prompt / settingTrained with frozen v1, 512pxTrained with frozen v2, 512pxRegularisation, no assistant
character_beach, seed 123character_beach, seed 123character_beach, seed 123character_beach, seed 123
character_beach, seed 777character_beach, seed 777character_beach, seed 777character_beach, seed 777
character_scarf, seed 123character_scarf, seed 123character_scarf, seed 123character_scarf, seed 123
character_scarf, seed 777character_scarf, seed 777character_scarf, seed 777character_scarf, seed 777
character_bicycle, seed 123character_bicycle, seed 123character_bicycle, seed 123character_bicycle, seed 123
character_bicycle, seed 777character_bicycle, seed 777character_bicycle, seed 777character_bicycle, seed 777
character_umbrella, seed 123character_umbrella, seed 123character_umbrella, seed 123character_umbrella, seed 123
character_umbrella, seed 777character_umbrella, seed 777character_umbrella, seed 777character_umbrella, seed 777
prior_fox, seed 123prior_fox, seed 123prior_fox, seed 123prior_fox, seed 123
prior_fox, seed 777prior_fox, seed 777prior_fox, seed 777prior_fox, seed 777

Assistant v2 plus synthetic regularisation

These two additional runs combine frozen assistant v2 with Domokun and synthetic regularisation at 512px or 1024px training resolution. Each completed 2,000 optimizer updates: 904 Domokun and 1,096 regularisation batches. At the included 1,000-update midpoint, the counts were 439 and 561. Both start from identical concept-adapter weights and use the same backend-choice sequence, BF16 AdamW at 1e-4, norm clipping 1.0, rank/alpha 32/32 and batch size 1. The 27 Domokun images use the literal 🟫 caption. Each run uses 834 synthetic images from its respective square-resolution bucket, so the synthetic image pools also differ between resolutions. Equal backend sampling weights do not guarantee exactly equal realized counts.

Regularisation semantics are assistant-aware: the detached target is base + frozen assistant v2, with only the trainable concept adapter disabled. The student is base + frozen assistant v2 + concept adapter at the same noisy inputs. GPU audits in both runs confirmed exact agreement with an independent assistant-only prediction, a difference from the bare-base prediction, and restoration of the student stack with the assistant frozen. This avoids using a bare-base target to train the concept adapter to compensate for the assistant. Audit results and batch counts are in training_provenance.json.

The original grids below contain base on the left, trained concept checkpoint on the right, with the assistant disabled. They use 40 steps, CFG 1 and CPU-generator seed 42. Here 512px and 1024px output refer to actual panel dimensions, independently of training resolution. The PNGs include labels and borders around those panels. These are single-seed periodic validations, not the earlier two-seed CFG-4 screen.

At the final checkpoint, both runs retain coherent scenes, portraits and fox illustrations in this selection. The 512px-trained model produces a recognisable brown character in the beach/scarf scenes and a character on a bicycle at both output sizes. Body proportions and bicycle interactions remain imperfect. The 1024px-trained model also produces recognisable beach/scarf characters, but its 1024px bicycle output resembles a human cyclist in a brown creature costume. Its scarf colour differs between output sizes. These concept/pose errors should be distinguished from overall image coherence; this small screen does not establish which preservation method is best.

VAE caveat: these images use the original pinned Qwen VAE, without tiling. Ollin’s texture-fix VAE was not used. Canvas/checkerboard texture should be investigated with an identical-latent decoder comparison before attributing it to an assistant or training resolution. No texture-fix results are implied by this section.

Expand: assistant v2 + regularisation, final checkpoint, 512px output
Prompt / setting512px training1024px training
character_beachcharacter_beachcharacter_beach
character_scarfcharacter_scarfcharacter_scarf
character_bicyclecharacter_bicyclecharacter_bicycle
prior_foxprior_foxprior_fox
prior_portraitprior_portraitprior_portrait
Expand: assistant v2 + regularisation, final checkpoint, 1024px output
Prompt / setting512px training1024px training
character_beachcharacter_beachcharacter_beach
character_scarfcharacter_scarfcharacter_scarf
character_bicyclecharacter_bicyclecharacter_bicycle
prior_foxprior_foxprior_fox
prior_portraitprior_portraitprior_portrait
Expand: assistant v2 + regularisation, midpoint to final, 1024px output
Prompt / settingUpdate 1,000Update 2,000
512px training, character_beach512px training, character_beach512px training, character_beach
512px training, character_bicycle512px training, character_bicycle512px training, character_bicycle
1024px training, character_beach1024px training, character_beach1024px training, character_beach
1024px training, character_bicycle1024px training, character_bicycle1024px training, character_bicycle

Both midpoint and final adapters are under checkpoints/assisted-v2-regularised-{512,1024}/step-{1000,2000}/. All 40 original validation grids for those checkpoints are included under assets/periodic/. Use only the concept adapter for inference.

The corresponding recipes require assistant-aware parent capture. The tested context implementation is supplied as recipes/assistant-aware-regularisation.py, with its hash in the provenance file. For a compatible SimpleTuner build without this fix, copy that file to the checkout root alongside the selected recipe in config/experiment/, then launch:

SIMPLETUNER_ENVIRONMENT=experiment SIMPLETUNER_CONFIG_BACKEND=json python -c 'import runpy; runpy.run_path("assistant-aware-regularisation.py"); runpy.run_module("simpletuner.train", run_name="__main__")'

This selects only the frozen assistant during parent capture and restores the training stack afterward. The supplied recipe files alone do not change an older trainer’s bare-base target semantics. These recipes start fresh adapters; the published files are not full optimizer checkpoints.

5. A useful failure: increasing the learning rate

An assisted 512px run at 3e-4, norm clip 1.0, severely degraded all four periodic prompts by update 500. The unrelated fox and elderly-woman portrait became Domokun-like imagery. Its checkpoint is included as a failure example, not a recommended adapter. This separate run is not a fully matched causal learning-rate study.

Expand: high-learning-rate failure at update 500
Prompt / settingBase / trained checkpoint
domo_beachdomo_beach
domo_scarfdomo_scarf
prior_foxprior_fox
prior_portraitprior_portrait

Reading and reproducing the comparisons

  • Periodic grids: the original saved images, with base and trained panels. CPU-generator seed 42, 40 steps, CFG 1. Assistant-v1 panels are 1024px; the earlier concept panels are 512px. The assistant-plus-regularisation section explicitly includes both 512px and 1024px panels. Treat them as training histories, not a uniform cross-run benchmark.
  • Independent inference: 512×512, 40 steps, GPU-generator seeds in filenames, adapter strength 1, full-frame VAE. CFG 4 means true conditional/unconditional passes with an empty negative prompt: 80 audited transformer forwards versus 40 at CFG 1. Same numeric seeds on CPU and CUDA do not imply the same initial noise.
  • Held-out screen: seeds 123 and 777, CFG 4, character-noun prompts. These prompts/seeds were used for inspection after initial prompt exploration; this is a small qualitative screen, not a statistically held-out benchmark.
  • Each variant has one training seed. Historical runs are not a complete factorial experiment; the source-image transforms/cache histories and prompt sets were not identical across every run. The paired v2 configs and initial concept weights are matched, but different training resolutions necessarily change the image/latent tensors.
  • The images shown are selected for practical questions, with multiple scenes and unrelated subjects retained. No quantitative quality score or general causal attribution is claimed. We have not established whether observed degradation specifically measures loss of CFG distillation.

The exact prompts are in prompts.json, file hashes and inference provenance in images.json, and selected adapter hashes in artifacts.json. Recorded training revisions and batch counts are in training_provenance.json. Additional saved inference images are available under assets, including the second seed and CFG settings where recorded.

Using the weights

Weights live under checkpoints/<experiment>/step-<N>/pytorch_lora_weights.safetensors. They are individual LoRA adapters, not full model checkpoints or resumable optimizer states. Only use a concept adapter for final Domokun inference; do not also load its training assistant.

Download a chosen artifact:

from huggingface_hub import hf_hub_download

adapter = hf_hub_download(
    "SimpleTuner/Qwen-Image-2.1-LoRA-experiments",
    "checkpoints/assisted-v2-multiscale-repa/step-4000/pytorch_lora_weights.safetensors",
)

For a new downstream training run with v2, add these fields to a complete SimpleTuner concept-training configuration:

{
  "assistant_lora_path": "SimpleTuner/Qwen-Image-2.1-training-assistant-v2",
  "disable_assistant_lora": false,
  "assistant_lora_strength": 1.0,
  "assistant_lora_inference_strength": 0.0
}

See recipes for configurations. To run one, copy its three JSON files into config/experiment/ in a SimpleTuner checkout with Qwen Image 2.1 and assistant-LoRA support, then run simpletuner train env=experiment. These are fresh-run recipes; the released weights do not contain optimizer/data-loader state for training resume. The evaluation script evaluation/render.py reads the same prompt-library location and can render the pinned base alone or a chosen local adapter directory. It requires a SimpleTuner build with the Qwen Image 2.1 and assistant-LoRA implementations used for these experiments. This repository contains adapters, configurations and evaluation code, not a packaged training runtime; the recorded wheel hash identifies the tested build but does not by itself reproduce that environment.

Data and license

The assistant-v1 weights match the corrected v1 release; this repository does not overwrite it. Assistant v2 uses real CC12M images with structured captions, real e621 images with Webshart indices, and the generated-image dataset. The e621 source can contain adult material; the displayed evaluation prompts are ordinary character and portrait tests. We do not redistribute the source training images here.

Concept data comes from RareConcepts/Domokun. These experiments make no claim of ownership over the character or source imagery. The base and adapter weights are subject to the upstream Qwen Research License Agreement. See Notice for attribution and the modification notice. Source dataset terms remain separate from the model license.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for SimpleTuner/Qwen-Image-2.1-LoRA-experiments

Adapter
(40)
this model

Datasets used to train SimpleTuner/Qwen-Image-2.1-LoRA-experiments