RunDiffusion
/

Wonderman-Flux-POC

@@ -63,8 +63,6 @@ widget:
 license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md
 ---
 <div style="display: flex; align-items: center; justify-content: space-between;"> <img src="https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Huggingface-assets/RD-Logo-dark.jpg" alt="Left Image" style="width: 30%;">
 <p style="text-align: center; width: 40%;">
   <span style="font-weight: bold; font-size: 1.5em;">Flux Training Concept - Wonderman POC</span><br>
@@ -92,7 +90,7 @@ Flux thinks that "Wonderman" is "Superman"
 ## Data Used for Training
-You can view the [RAW low quality data here: ](https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/tree/main/Raw%20Low%20Quality%20Data).
 The training data was low resolution, cropped, oddly shaped, pixelated, and overall the worst possible data we've come across. That didn't stop us! AI to the rescue!
 ![Low Quality Training Data](Huggingface-assets/multiple-samples-training-data.png)
@@ -114,191 +112,95 @@ A vintage comic book cover of Wonderman. On the cover, there are three main char
 ![Vintage Wonderman](Cleaned%20and%20Captioned%20Data/00008.png)
 Wonderman, a male superhero character. He is wearing a green and red costume with a large 'W' emblem on the chest. Wonderman has a muscular physique, brown hair, and is wearing a black mask covering his eyes. He stands confidently with his hands by his sides. photo
-![Standing Wonderman](https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Cleaned%20and%20Captioned%20Data/00002.png)
 ### Train the Data
 All tasks were performed on a local workstation equipped with an RTX 4090, i7 processor, and 64GB RAM. Note that 32GB RAM will not suffice, as you may encounter out-of-memory (OOM) errors when caching latents. We did use RunDiffusion.com for testing the LoRAs created, enabling us to launch five servers with five checkpoints to determine the best one that converged
 We're not going to dive into the rank and learning rate and stuff because this really depends on your goals and what you're trying to accomplish. But the rules below are good ones to follow.
-- We used Ostris's ai-toolkit available here: https://github.com/ostris/ai-toolkit/tree/main
-- Default config with LR: 4e-4 at Rank 16
 - 2200 - 2600 steps saw good convergence. Even some checkpoints into the 4k step range turned out pretty good.
 If targeting finer details, you may want to adjust the rank up to 32 and lower the learning rate. You will also need to run more steps if you do this.
 **Training a style:** Using simple captions with clear examples to maintain a coherent style is crucial. Although caption-less LoRAs can sometimes work for styles, this was not within the scope of our goals, so we cannot provide specific insights.
 **Training a concept:** You can choose either descriptive captions to avoid interfering with existing tokens or general captions that might interfere, depending on your intention. This choice should be intentional.
 Captioning has never been more critical. Flux "gives you what you ask for" - and that's a good thing. You can train a LoRA on a single cartoon concept and still generate photo realistic people. You can even caption a cartoon in the foreground and a realistic scene in the background! This capability is BY DESIGN - so do not resist it - embrace it! (Spoiler alert next!)
-![prompt different backgrounds]()
 You'll see in the next page of examples where the captioning really helps or hurts you. Depending on your goals again you will need to choose the path that fits what you're trying to accomplish.
 Total time for the LoRA was about 2 to 2.5 hours. $1 to $2 on RunPod, Vast, or local electricity will be even cheaper.
 Now for the results! (This next file is big to preserve the quality)
-## 500 Steps
 ![500 steps](Huggingface-assets/500-steps.jpg)
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]
 - **Developed by:** Darin Holbrook - RunDiffusion co-founder and Chief Technology Officer
 - **Funded by:** RunDiffusion.com / RunPod.io
 - **Model type:** Flux [dev] LoRA

 license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md
 ---
 <div style="display: flex; align-items: center; justify-content: space-between;"> <img src="https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Huggingface-assets/RD-Logo-dark.jpg" alt="Left Image" style="width: 30%;">
 <p style="text-align: center; width: 40%;">
   <span style="font-weight: bold; font-size: 1.5em;">Flux Training Concept - Wonderman POC</span><br>
 ## Data Used for Training
+You can view the [RAW low quality data here: ](https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/tree/main/Raw%20Low%20Quality%20Data)
 The training data was low resolution, cropped, oddly shaped, pixelated, and overall the worst possible data we've come across. That didn't stop us! AI to the rescue!
 ![Low Quality Training Data](Huggingface-assets/multiple-samples-training-data.png)
 ![Vintage Wonderman](Cleaned%20and%20Captioned%20Data/00008.png)
 Wonderman, a male superhero character. He is wearing a green and red costume with a large 'W' emblem on the chest. Wonderman has a muscular physique, brown hair, and is wearing a black mask covering his eyes. He stands confidently with his hands by his sides. photo
+![Standing Wonderman](Cleaned%20and%20Captioned%20Data/00002.png)
 ### Train the Data
 All tasks were performed on a local workstation equipped with an RTX 4090, i7 processor, and 64GB RAM. Note that 32GB RAM will not suffice, as you may encounter out-of-memory (OOM) errors when caching latents. We did use RunDiffusion.com for testing the LoRAs created, enabling us to launch five servers with five checkpoints to determine the best one that converged
 We're not going to dive into the rank and learning rate and stuff because this really depends on your goals and what you're trying to accomplish. But the rules below are good ones to follow.
+- We used Ostris's ai-toolkit available here: [Ostris ai-toolkit](https://github.com/ostris/ai-toolkit/tree/main)
+- Default config with LR 4e-4 at Rank 16
 - 2200 - 2600 steps saw good convergence. Even some checkpoints into the 4k step range turned out pretty good.
 If targeting finer details, you may want to adjust the rank up to 32 and lower the learning rate. You will also need to run more steps if you do this.
 **Training a style:** Using simple captions with clear examples to maintain a coherent style is crucial. Although caption-less LoRAs can sometimes work for styles, this was not within the scope of our goals, so we cannot provide specific insights.
 **Training a concept:** You can choose either descriptive captions to avoid interfering with existing tokens or general captions that might interfere, depending on your intention. This choice should be intentional.
 Captioning has never been more critical. Flux "gives you what you ask for" - and that's a good thing. You can train a LoRA on a single cartoon concept and still generate photo realistic people. You can even caption a cartoon in the foreground and a realistic scene in the background! This capability is BY DESIGN - so do not resist it - embrace it! (Spoiler alert next!)
+![prompt different backgrounds](Huggingface-assets/cartoon-foreground-realistic-background.jpg)
 You'll see in the next page of examples where the captioning really helps or hurts you. Depending on your goals again you will need to choose the path that fits what you're trying to accomplish.
 Total time for the LoRA was about 2 to 2.5 hours. $1 to $2 on RunPod, Vast, or local electricity will be even cheaper.
 Now for the results! (This next file is big to preserve the quality)
+# 500 Steps
+Right off the bat at 500 steps you will get some likeness. This will mostly be baseline Flux. If you're training a concept that exists then you will see some convergence even at just 500 steps.
 ![500 steps](Huggingface-assets/500-steps.jpg)
+Prompt: a vintage comic book cover for Wonderman, featuring three characters in a dynamic action scene. The central figure is Wonderman with a confident expression, wearing a green shirt with a yellow belt and red gloves. To his left is a woman with a look of concern, dressed in a yellow top and red skirt. On the right, there's a monstrous creature with sharp teeth and claws, seemingly attacking the man. The background is minimal, primarily blue with a hint of landscape at the bottom. The text WONDER COMICS and No. 11 suggests this is from a series.
+# 1250 Steps
+It will start to break apart a little bit here. Be patient. It's learning.
+![1250 steps](Huggingface-assets/1250-steps.jpg)
+Prompt: A vintage comic book cover titled 'Wonderman Comics'. The central figure is Wonderman who appears to be in a combat stance. He is lunging at a large, menacing creature with a gaping mouth, revealing sharp teeth. Below the main characters, there's a woman in a yellow dress holding a small device, possibly a gun. She seems to be in distress. In the background, there's a futuristic-looking tower with a few figures standing atop. The overall color palette is vibrant, with dominant yellows, greens, and purples.
+# 1750 Steps
+Hey! We're getting somewhere! The caption as a prompt should be showing our subject well at this stage but the real test is breaking away from the caption to see if our subject is present.
+![1750 steps](Huggingface-assets/1750-steps.jpg)
+Prompt: Wonderman wearing a green and red costume with a large 'W' emblem on the chest standing heroically
+# 2500 Steps
+There he is! We can now prompt more freely to get Wonderman doing other stuff. Keep in mind we will still be limited to what we trained on, but at least we have a great starting point!
+![2500 steps](Huggingface-assets/2500-steps.jpg)
+Prompt: comic style illustration of Wonderman running from aliens on the moon. center character is Wonderman, a male superhero character. He is wearing a green and red costume with a large 'W' emblem on the chest. Black boots to his knees. Wonderman is wearing a black mask covering his eyes
+<div style="display: grid; grid-template-columns: repeat(auto-fill, minmax(300px, 1fr)); gap: 10px;">
+  <div>
+    <img src="https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Huggingface-assets/Cosplay-realistic.jpg" alt="Image 1" style="width: 100%;">
+    <p style="text-align: center;">"Cosplay" always get's super heroes to appear realistic</p>
+  </div>
+  <div>
+    <img src="https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Huggingface-assets/anime-sample.jpg" alt="Image 2" style="width: 100%;">
+    <p style="text-align: center;">Anime Style</p>
+  </div>
+  <div>
+    <img src="https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Huggingface-assets/cosplay-realistic-2.jpg" alt="Image 3" style="width: 100%;">
+    <p style="text-align: center;">Photo Realistic</p>
+  </div>
+  <div>
+    <img src="https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Huggingface-assets/comic-book-style-moon.jpg" alt="Image 3" style="width: 100%;">
+    <p style="text-align: center;">Illustration Style</p>
+  </div>
+</div>
+# Conclusion
+This proof of concept provided valuable insights into working with Flux. One of the key lessons we took away is that while Flux is straightforward to train, it's crucial to clearly define your objectives before diving in. Without a clear vision, you might find your model either overwhelming or underwhelming—especially with concepts like "a cookie," which already has extensive representation within Flux.
+Every training project comes with its own distinct set of challenges and complexities. Patience and persistence are essential to navigate these obstacles successfully. With careful planning and a focused approach, you'll be able to overcome these hurdles and achieve your desired outcomes.
+## Things We Would Do Different
+Upon reviewing our example data, we identified several areas that could benefit from additional cleanup. These issues impacted the final model, leading to some unexpected results. For instance, when "Wonderman" was prompted, the model occasionally generated elements of "Superman" due to similarities between the two. This led to the appearance of a "cape" in an lot of generations. Another issue we found was the appearance of multicolored tights, with some samples showing red while others displayed green. Additionally, the model produced purple shorts again, which was a direct result of the training data.
+While these challenges surfaced during the process, we believe they can be resolved with further refinement and adjustment of the dataset. Addressing these inconsistencies will improve the accuracy and quality of the model output.
+<div style="display: grid; grid-template-columns: repeat(auto-fill, minmax(300px, 1fr)); gap: 10px;">
+  <div>
+    <img src="https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Huggingface-assets/wonderman-superman-mix.jpg" alt="Image 1" style="width: 100%;">
+    <p style="text-align: center;">"Wonderman running from an alien on the moon with the earth in the sky in the background" and sometimes we get "Superman"</p>
+  </div>
+  <div>
+    <img src="https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Huggingface-assets/wonderman-cape.jpg" alt="Image 2" style="width: 100%;">
+    <p style="text-align: center;">"Cape" is present even though it was not prompted</p>
+  </div>
+  <div>
+    <img src="https://huggingface.co/RunDiffusion/Wonderman-Flux-POC/resolve/main/Huggingface-assets/wonderman-two-colors-tights.jpg" alt="Image 3" style="width: 100%;">
+    <p style="text-align: center;">Red and green tights were both present due to the training data which had samples of both.</p>
+  </div>
+</div>
+[View all the generations here](Wonderman%20Generation%20Samples)
+# Special Thanks
+The team had a blast with this project and we can't wait to start the next one. The Wonderman LoRA will be available for download on Huggingface and Civitai for research. At RunDiffusion we are always trying to push the boundaries with whatever bleeding-edge tech comes out. We appreciate all our customers and supporters. Without you we would not have the funds to dedicate a team to research, so thank you!
+- **Ostris**: [Thank you to Ostris](https://x.com/ostrisai) for the awesome training tools!
+- **Mint**: [Thank you to Mint](https://x.com/araminta_k) for the awesome YouTube tutorial!
+- **RunPod**: For the compute credit
+# More Credits
 - **Developed by:** Darin Holbrook - RunDiffusion co-founder and Chief Technology Officer
 - **Funded by:** RunDiffusion.com / RunPod.io
 - **Model type:** Flux [dev] LoRA