control-edgedrawing-cv480edpf-drop50-fp16-checkpoint-45000

README.md

@@ -1,20 +1,20 @@
 ---
-license: openrail
+license: cc-by-nc-sa-4.0
 datasets:
 - ChristophSchuhmann/improved_aesthetics_6.5plus
 language:
 - en
 ---
 
-Controls image generation by edge maps generated with [EdgeDrawing Parameter-Free](https://github.com/CihanTopal/ED_Lib).
+Controls image generation by edge maps generated with [Edge Drawing](https://github.com/CihanTopal/ED_Lib). Edge Drawing comes in different flavors: original (experiments 1-2), parameter-free (experiments 3+), color (not yet available).
 
-* Based on my GitHub monologs at [Edge Drawing - a Canny alternative](https://github.com/lllyasviel/ControlNet/discussions/318)
-* For usage see the model page on [Civitai.com](https://civitai.com/models/149740).
-* To generate edpf maps you can use the script [edpf.py](https://gitlab.com/-/snippets/3601881).
+* Based on my monologs at [github.com - Edge Drawing](https://github.com/lllyasviel/ControlNet/discussions/318)
+* For usage see the model page on [civitai.com - Model](https://civitai.com/models/149740).
+* To generate edpf maps you can use the script [gitlab.com - edpf.py](https://gitlab.com/-/snippets/3601881).
 * For evaluation see the corresponding .zip with images in "files".
-* To run your own evaluations you can use [inference.py](https://gitlab.com/-/snippets/3602096).
+* To run your own evaluations you can use the script [gitlab.com - inference.py](https://gitlab.com/-/snippets/3602096).
 
-**EdgeDrawing Parameter-Free**
+**Edge Drawing Parameter Free**
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64c0ec65a2ec8cb2f589233a/jmdCGeMJx4dKFGo44cuEq.png)
 
@@ -26,10 +26,12 @@ prompt: _a detailed high-quality professional photo of swedish woman standing in
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64c0ec65a2ec8cb2f589233a/2PSWsmzLdHeVG-i67S7jF.png)
 
-**Canndy Edge Detection (default in Automatic1111)**
+**Canndy Edge for comparison (default in Automatic1111)**
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/64c0ec65a2ec8cb2f589233a/JZTpa-HZfw0NUYnxZ52Iu.png)
 
+_notice all the missing edges, the noise and artifacts. yuck! ugh!_
+
 # Image dataset
 
 * [laion2B-en aesthetics>=6.5 dataset](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6.5plus)
@@ -55,13 +57,46 @@ accelerate launch train_controlnet.py ^
   --seed=0
 ```
 
+# Evaluation
+
+To evaluate the model it makes sense to compare it with the original Canny model. Original evaluations and comparisons are available at [ControlNet 1.0 repo](https://github.com/lllyasviel/ControlNet), [ControlNet 1.1 repo](https://github.com/lllyasviel/ControlNet-v1-1-nightly), [ControlNet paper v1](https://arxiv.org/abs/2302.05543v1), [ControlNet paper v2](https://arxiv.org/abs/2302.05543) and [Diffusers implementation](https://huggingface.co/takuma104/controlnet_dev/tree/main). Some points we have to keep in mind when comparing canny with edpf in order not to compare apples with oranges:
+* canny 1.0 model was trained on 3M images, canny 1.1 model on even more, while edpf model so far is only trained on a 180k-360k.
+* canny edge-detector requires parameter tuning while edpf is parameter-free.
+* Do we manually fine-tune canny to find the perfect input image or do we leave it at default? We could argue that "no fine-tuning required" is the usp of edpf and we want to compare in the default setting, whereas canny fine-tuning is subjective.
+* Would the canny model actually benefit from a edpf pre-processor and we might not even require a edpf model?
+* When evaluating human images we need to be aware of Stable Diffusion's inherent limits, like disformed faces and hands.
+* When evaluating style we need to be aware of the bias from the image dataset (laion2b-en-aesthetics65), which might tend to generate "aesthetic" images, and not actually work "intrisically better".
+
 # Versions
 
-**Experiment 5 - control-edgedrawing-cv480edpf-drop0+50-fp16-checkpoint-118000**
+**Experiment 1 - 2023-09-19 - control-edgedrawing-default-drop50-fp16-checkpoint-40000**
+
+Images converted with https://github.com/shaojunluo/EDLinePython (based on original (non-parameter free) edge drawing). Default settings are:
+
+`smoothed=False`
+
+```
+{ 'ksize'            :  5
+, 'sigma'            :  1.0
+, 'gradientThreshold': 36
+, 'anchorThreshold'  :  8
+, 'scanIntervals'    :  1
+}
+```
+
+additional arguments: `--proportion_empty_prompts=0.5`.
 
-see experiment 4. resumed with epoch 2 from 90000 using `--proportion_empty_prompts=0.5` => results became worse, CN didn't pick up on no-prompts (I also tried checkpoint-104000). restarting with 50% drop.
+Trained for 40000 steps with default settings => empty prompts were probably too excessive
 
-**Experiment 4 - control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-90000**
+Update 2023-09-22: bug in algorithm produces too sparse images on default, see https://github.com/shaojunluo/EDLinePython/issues/4
+
+**Experiment 2 - 2023-09-20 - control-edgedrawing-default-noisy-drop0-fp16-checkpoint-40000**
+
+Same as experiment 1 with `smoothed=True` and `--proportion_empty_prompts=0`.
+
+Trained for 40000 steps with default settings => conditioning images are too noisy
+
+**Experiment 3.0 - 2023-09-22 - control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-45000**
 
 Conditioning images generated with [edpf.py](https://gitlab.com/-/snippets/3601881) using [opencv-contrib-python::ximgproc::EdgeDrawing](https://docs.opencv.org/4.8.0/d1/d1c/classcv_1_1ximgproc_1_1EdgeDrawing.html).
 
@@ -74,38 +109,31 @@ edges    = ed.detectEdges(image)
 edge_map = ed.getEdgeImage(edges)
 ``` 
 
-90000 steps (45000 steps on original, 45000 steps with left-right flipped images)
+45000 steps => This is **version 0.1 on civitai**.
 
-**Experiment 3 - control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-45000**
+**Experiment 3.1 - 2023-09-24 - control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-90000**
 
-see experiment 4. 45000 steps. This is version 0.1 on civitai.
+90000 steps (45000 steps on original, 45000 steps with left-right flipped images) => quality became better, might release as 0.2 on civitai.
 
-**Experiment 2 - control-edgedrawing-default-noisy-drop0-fp16-checkpoint-40000**
-
-Images converted with https://github.com/shaojunluo/EDLinePython
-
-Default settings are:
-
-`smoothed=False`
+**Experiment 3.2 - 2023-09-24 -control-edgedrawing-cv480edpf-drop0+50-fp16-checkpoint-118000**
 
-```
-{ 'ksize'            :  5
-, 'sigma'            :  1.0
-, 'gradientThreshold': 36
-, 'anchorThreshold'  :  8
-, 'scanIntervals'    :  1
-}
-```
+resumed with epoch 2 from 90000 using `--proportion_empty_prompts=0.5` => results became worse, CN didn't pick up on no-prompts (I also tried intermediate checkpoint-104000). restarting with 50% drop.
 
-`smoothed=True`, but no empty prompts. Trained for 40000 steps with default settings => conditioning images are too noisy.
+**Experiment 4.0 - 2023-09-25 - control-edgedrawing-cv480edpf-drop50-fp16-checkpoint-45000**
 
-**Experiment 1 - control-edgedrawing-default-drop50-fp16-checkpoint-40000**
+see experiment 3.0. restarted from 0 with `--proportion_empty_prompts=0.5` => 
 
-Same as experiment 2.
+**Experiment 4.1 - control-edgedrawing-cv480edpf-drop50-fp16-checkpoint-45000**
 
-Update: bug in algorithm produces too sparse images on default, see https://github.com/shaojunluo/EDLinePython/issues/4
+# Ideas
 
-additional arguments: `--proportion_empty_prompts=0.5`. Trained for 40000 steps with default settings => empty prompts were probably too excessive
+* fine-tune off canny
+* cleanup image dataset (l65)
+* uncropped mod64 images
+* integrate edcolor
+* bigger image dataset (gcc)
+* cleanup image dataset (gcc)
+* re-train with fp32
 
 # Question and answers
 

config.json

@@ -0,0 +1,51 @@
+{
+  "_class_name": "ControlNetModel",
+  "_diffusers_version": "0.22.0.dev0",
+  "_name_or_path": "control-edgedrawing-pf-drop0-fp16/checkpoint-71000",
+  "act_fn": "silu",
+  "addition_embed_type": null,
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": null,
+  "attention_head_dim": 8,
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "class_embed_type": null,
+  "conditioning_channels": 3,
+  "conditioning_embedding_out_channels": [
+    16,
+    32,
+    96,
+    256
+  ],
+  "controlnet_conditioning_channel_order": "rgb",
+  "cross_attention_dim": 768,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "global_pool_conditions": false,
+  "in_channels": 4,
+  "layers_per_block": 2,
+  "mid_block_scale_factor": 1,
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_time_scale_shift": "default",
+  "transformer_layers_per_block": 1,
+  "upcast_attention": false,
+  "use_linear_projection": false
+}

control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-45000.zip

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:16d4d36d3a577b1fa18a5f07a80d1bfc8ca10f1461d8c7e45d79f7285d496b3c
-size 17597750
+oid sha256:431d99dc147ab50c2672093f217d48d1f3735d071c471d4a017559bffe013cd9
+size 5582414

control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-90000.zip

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fb2349062e57cbf5d25e249760f6893c20fb34fb099a98378496e7105f91f07f
-size 16943930
+oid sha256:fdb6844d462fd0d60fd4f57cd8de07f6b2861f218f2a415ff26f2271c96cd044
+size 4928594

control-edgedrawing-cv480edpf-drop50-fp16-checkpoint-45000.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:322df64c934f6ffd7fd58ad79baa07aa8de34b42f3d7b777e77ffb915cdf1808
+size 722598616

control-edgedrawing-cv480edpf-drop50-fp16-checkpoint-45000.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efd2831e49f4af3564291cb7db60dd1d73b9afd87e734e8e802f8467a6133869
+size 5777208

eval_canny_canny.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:193a684863a49d0f9a2b2d89b930483374ff9d472af174093823336c63c977ad
+size 4512344

eval_canny_edpf.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94fb8603912563a8ccab5efe58bbe7105829f7619cbeff10330429ac23b337d9
+size 3887066

eval_input.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9c509718d6c0cc0e234c3e402a6f632afde6d9bdd27a49afb7b25246a4ec1df1
+size 3617968