license: openrail
datasets:
- ChristophSchuhmann/improved_aesthetics_6.5plus
language:
- en
Based on my GitHub monologs at Edge Drawing - a Canny alternative
Controls image generation by edge maps generated with EdgeDrawing Parameter-Free.
For usage see the model page on Civitai.com. For evaluation see the corresponding .zip files with images. To run your own evaluation you can use inference.py.
EdgeDrawing Parameter-Free
Example
sampler=UniPC steps=20 cfg=7.5 seed=0 batch=9 model: v1-5-pruned-emaonly.safetensors cherry-picked: 1/9
prompt: a detailed high-quality professional photo of swedish woman standing in front of a mirror, dark brown hair, white hat with purple feather
Canndy Edge Detection (default in Automatic1111)
Image dataset
- laion2B-en aesthetics>=6.5 dataset
--min_image_size 512 --max_aspect_ratio 2 --resize_mode="center_crop" --image_size 512- resulting in 180k images
Training
accelerate launch train_controlnet.py ^
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" ^
--output_dir="control-edgedrawing-[version]-fp16/" ^
--dataset_name="mydataset" ^
--mixed_precision="fp16" ^
--resolution=512 ^
--learning_rate=1e-5 ^
--train_batch_size=1 ^
--gradient_accumulation_steps=4 ^
--gradient_checkpointing ^
--use_8bit_adam ^
--enable_xformers_memory_efficient_attention ^
--set_grads_to_none ^
--seed=0
Versions
Experiment 5 - control-edgedrawing-cv480edpf-drop0+50-fp16-checkpoint-118000
see experiment 4. resumed with epoch 2 from 90000 using --proportion_empty_prompts=0.5 => results became worse, CN didn't pick up on no-prompts (I also tried checkpoint-104000). restarting with 50% drop.
Experiment 4 - control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-90000
Conditioning images generated with edpf.py using opencv-contrib-python::ximgproc::EdgeDrawing.
ed = cv2.ximgproc.createEdgeDrawing()
params = cv2.ximgproc.EdgeDrawing.Params()
params.PFmode = True
ed.setParams(params)
edges = ed.detectEdges(image)
edge_map = ed.getEdgeImage(edges)
90000 steps (45000 steps on original, 45000 steps with left-right flipped images)
Experiment 3 - control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-45000
see experiment 4. 45000 steps. This is version 0.1 on civitai.
Experiment 2 - control-edgedrawing-default-noisy-drop0-fp16-checkpoint-40000
Images converted with https://github.com/shaojunluo/EDLinePython
Default settings are:
smoothed=False
{ 'ksize' : 5
, 'sigma' : 1.0
, 'gradientThreshold': 36
, 'anchorThreshold' : 8
, 'scanIntervals' : 1
}
smoothed=True, but no empty prompts. Trained for 40000 steps with default settings => conditioning images are too noisy.
Experiment 1 - control-edgedrawing-default-drop50-fp16-checkpoint-40000
Same as experiment 2.
Update: bug in algorithm produces too sparse images on default, see https://github.com/shaojunluo/EDLinePython/issues/4
additional arguments: --proportion_empty_prompts=0.5. Trained for 40000 steps with default settings => empty prompts were probably too excessive
Question and answers
Q: What's the point of another edge control net anyway?
A: 🤷