--- license: apache-2.0 datasets: - KBlueLeaf/danbooru2023-webp-4Mpixel - KBlueLeaf/danbooru2023-metadata-database base_model: - black-forest-labs/FLUX.1-schnell - FA770/Sumeshi_Flux.1_S_v002E pipeline_tag: text-to-image tags: - anime - girls --- ![sample_image](./sample_images/1.webp) # Model Information **Note:** This model is a Schnell-based model, but it requires CFG scale of 3 or higher (not guidance scale) with 20 steps or more. It needs to be used with `clip_l_nigemono_flux1C`. **At this time, this model cannot be generated in WebUI Forge. Please use it with ComfyUI.** My English is terrible, so I use translation tools. ## Description Nigemono_Flux.1_Compact7.5B is an experimental anime model to verify the reduction of parameters in the Flux model. The number of parameters has been reduced from `12B` (double blocks 19/ single blocks 38) to `7.5B` (double blocks 12/ single blocks 24). You can use a negative prompt which works to some extent. The output is blurred and the style varies depending on the prompt, perhaps because the model has not been fully trained. ## Usage - Resolution: Like other Flux models - **(Distilled) Guidance Scale:** (Distilled) Guidance Scale: 0 (Does not work due to Schnell-based model) - **CFG Scale:** 6 ~ 9 (recommend 7; scale 1 does not generate decent outputs) - **Steps:** 20 ~ 100 or and more (recommend 40) - sampler: Euler - scheduler: Simple, Beta ## Prompt Format (from [Kohaku-XL-Epsilon](https://huggingface.co/KBlueLeaf/Kohaku-XL-Epsilon)) ```<1girl/1boy/1other/...>, , , , , , , , ``` Due to the small amount of training, the `` tags are almost non-functional. As training is focused on girl characters, it may not generate boy or other non-persons well. Since the dataset was created using hakubooru, the prompt format will be the same as the KohakuXL format. However, based on experiments, it is not strictly necessary to follow this format, as it interprets meaning to some extent even in natural language. ### Special Tags - **Quality Tags:** masterpiece, best quality, great quality, good quality, normal quality, low quality, worst quality - **Rating Tags:** safe, sensitive, nsfw, explicit - **Date Tags:** newest, recent, mid, early, old ## Training ### Dataset Preparation I used [hakubooru](https://github.com/KohakuBlueleaf/HakuBooru)-based custom scripts. - **Exclude Tags:** `traditional_media, photo_(medium), scan, animated, animated_gif, lowres, non-web_source, variant_set, tall image, duplicate, pixel-perfect_duplicate` - **Minimum Post ID:** 1,000,000 ### Blocks Reduction I reduced the parameters by gradually removing blocks and retraining, so as not to completely disrupt the model's generation capability. Removing too many blocks at once results in completely noisy output, requiring retraining from scratch. - **Training Hardware:** A single RTX 4090 - **Method:** LoRA training and merging the results - **Training Script:** [sd-scripts](https://github.com/kohya-ss/sd-scripts) - **Basic Settings:** ```powershell accelerate launch --num_cpu_threads_per_process 4 flux_train_network.py --network_module networks.lora_flux --sdpa --gradient_checkpointing --cache_latents --cache_latents_to_disk --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk --max_data_loader_n_workers 1 --save_model_as "safetensors" --mixed_precision "bf16" --fp8_base --save_precision "bf16" --full_bf16 --min_bucket_reso 384 --max_bucket_reso 1536 --seed 1 --max_train_epochs 1 --keep_tokens_separator "|||" --network_dim 32 --network_alpha 32 --unet_lr 1e-4 --train_batch_size 2 --gradient_accumulation_steps 3 --optimizer_type adamw8bit --lr_scheduler="constant_with_warmup" --lr_warmup_steps 500 --vae_batch_size 4 --cache_info --guidance_scale 1 --timestep_sampling shift --model_prediction_type raw --discrete_flow_shift 1.8 --loss_type l2 --highvram --network_args "in_dims=[16,16,16,0,16]" --network_train_unet_only --bucket_no_upscale ``` --Continued training from sumeshi flux.1s v002E-- 1. remove double blocks 18 / single blocks 34,35,36,37 / guidance_in (sumeshi flux.1s 002E has zero tensor) 2. 3,893images (res1024 bs2 /res512 bs8 acc2 warmup50 --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles 1 --discrete_flow_shift 1.8) 4epochs 3. merged into model 4. remove double blocks 16,17 / single blocks 32,33 5. 3,893images (res1024 bs2 /res512 bs8 acc1 warmup50 --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles 2 --discrete_flow_shift 1.8) 2epochs 6. merged into model 7. remove double blocks 14,15 / single blocks 28,29,30,31 8. 3,893images (res1024 bs2 /res512 bs8 acc2 warmup50 --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles 2 --discrete_flow_shift 1.8) 2epochs 9. merged into model 10. remove double blocks 12,13 / single blocks 24,25,26,27 11. 3,893images (res1024 bs2 /res512 bs8 acc2 warmup50 --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles 2 --discrete_flow_shift 1.8) 2epochs 12. merged into model 13. 3,893images (Full-finetuned res1024 bs1 acc1 afafactor --optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False" lr5e-6 warmup100 max_grad_norm 0.0 discrete_flow_shift 2) 2epochs 14. 12,000images (res1024 bs2 acc5 warmup100 --discrete_flow_shift 2) 1epochs 15. 12,000images (res1024 bs2 acc3 warmup100 --discrete_flow_shift 2) 2epochs 16. merged into model 17. 12,000images (res1024 bs2 acc3 warmup100 --discrete_flow_shift 2.5) 3epochs 18. 12,000images (res1024 bs2 acc3 warmup100 --discrete_flow_shift 2.5) 3epochs 19. 12,000images (res512 bs8 acc1 warmup100 timestep_sampling flux_shift(fixed ver.) ) 12epochs 20. merged into model 21. 12,000images (res512 bs4 acc1 warmup100 timesteps_sampling sigmoid sigmoid_scale 0.75) 7epochs 22. 3,893images (res512 bs4 acc1 warmup100 timesteps_sampling sigmoid sigmoid_scale 0.6 --caption_dropout_rate 0.1) 16epochs 23. merged into model 24. 3,893images (res1024 bs2 / res512 bs4 acc2 warmup100 unet_lr1e-4 te_lr5e-5 timesteps_sampling sigmoid sigmoid_scale 0.6 --caption_dropout_rate 0.1) 6epochs 25. merged into model and CLIP_L ## Resources (License) - **FLUX.1-schnell (Apache2.0)** - **danbooru2023-webp-4Mpixel (MIT)** - **danbooru2023-metadata-database (MIT)** ## Acknowledgements - **black-forest-labs:** Thanks for publishing a great open source model. - **kohya-ss:** Thanks for publishing the essential training scripts and for the quick updates. - **Kohaku-Blueleaf:** Thanks for the extensive publication of the scripts for the dataset and the various training conditions.