Update README.md

README.md

@@ -196,6 +196,160 @@ Among the remaining ones LoRa and Dadaption are probably slightly better. This c
 ![xyz_grid-0000-20230327073826](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples/xyz_grid-0000-20230327073826.png)
 
 
+### A Certain Theory on Lora Transfer
+
+Inspired by the introduction of [AnyLora](https://civitai.com/models/23900/anylora) and an experiment done by @Machi#8166. I decided to further investigate the influence of base model. It is worth noticing that you may get different results due to difference in dataset and hyperparameters, though I do believe the general trend should be preserved. However, I do get some result that is contradictory to what Machi observed. In particular, counterfeit does not seem to be a good base model to train on in my experiments. Now, let's dive in.
+
+**I only put jpgs below, I invite you to download pngs from https://huggingface.co/alea31415/LyCORIS-experiments/tree/main/generated_samples_0329 to check the details yourself**
+
+#### Setup
+
+I used the default setup described earlier except that I set clip skip to 2 here to be coherent with what the creator of AnyLora suggests.
+I train on the following 15 models
+
+1. NAI
+2. [AnyLora](https://civitai.com/models/23900/anylora)
+3. [NeverEndingDream](https://civitai.com/models/10028) (NED)
+4. [Counterfeit-V2.5](https://civitai.com/models/10443/counterfeit-v25-25d-tweak)
+5. [Aom3a1](https://civitai.com/models/9942/abyssorangemix3-aom3)
+6. [Anything4.5](https://huggingface.co/andite/anything-v4.0/blob/main/anything-v4.5-pruned-fp16.ckpt)
+7. [ACertainty](https://huggingface.co/JosephusCheung/ACertainty) (AC)
+8. [BP](https://huggingface.co/Crosstyan/BPModel/blob/main/bp_mk5.safetensors)
+9. [NMFSAN](https://huggingface.co/Crosstyan/BPModel/blob/main/NMFSAN/README.md)
+10. [VBP2.2](https://t.me/StableDiffusion_CN/905650)
+11. [MyneFactoryBaseV1.0](https://civitai.com/models/9106/myne-factory-base) (MFB)
+12. [RacyMix](https://civitai.com/models/15727/racymix)
+13. [PastelMix](https://civitai.com/models/5414/pastel-mix-stylized-anime-model-fantasyai)
+14. [ChilloutMix](https://civitai.com/models/6424/chilloutmix)
+
+In addition to the above we test on the following models
+
+15. [Mix-Pro-V3](https://civitai.com/models/7241/mix-pro-v3)
+16. [Fantasy Background](https://civitai.com/?query=Fantasy%20Background)
+17. [Salt Mix](https://huggingface.co/yatoyun/Salt_mix)
+18. nep (the model is not online anymore)
+
+As we will see later, it seems that the above models can be roughly separate into the following groups
+- Source (Ancestor) Models: NAI, AC
+- Anything/Orange family: AOM, Anything, AnyLora, Counterfeit, NED
+- Photorealistic: NeverEndingDream, ChilloutMix
+- Pastel: RacyMix, PastelMix
+- AC family: ACertainty, BP, NMFSAN (BP is trained from AC and NMFSAN is trained from BP)
+- Outliers: the remaining ones, and notably MFB
+
+#### Character Training
+
+Most people who train characters want to be able to switch style by switching models. In this case you indeed want to use NAI, or otherwise the mysterious ACertainty
+
+**Anisphia**
+![xyz_grid-0007-20230330035309](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0007-20230330035309.jpg)
+
+**Tilty**
+![xyz_grid-0011-20230330053814](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0011-20230330053814.jpg)
+
+Character training is mostly robust to model transfer. Nonetheless, if you pick models that are far away, such as MFB, or pastel-mix, to train on, the results could be much worse.
+
+Of course you can have two characters in the same image as well.
+
+**Anisphia X Euphyllia**
+![xyz_grid-0017-20230330074416](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0017-20230330074416.jpg)
+
+
+#### Style Training
+
+In style training people may want to retain the styles when switching model but instead benefit from other components of the base model (such as more detailed background). Unfortunately, it seems that there is not a single go-to model for style training. The general trend is, you get better style preservation if the two models are close enough, and in particular if they are in the same group described above. This is actually already clear with the anime styles in the previous images. Let us check some more specific examples.
+
+**ke-ta**
+![xyz_grid-0004-20230330021202](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0004-20230330021202.jpg)
+
+**onono imoko**
+![xyz_grid-0006-20230330033041](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0006-20230330033041.jpg)
+
+A lot of people like to use models of the orange and anything series. In this case, as claimed AnyLora can indeed be a good base model for style training. However, training on Anything, AOM, or even NED could also be more or less effective.
+
+It is also worth noticing that while training on PastelMix and RacyMix give bad results on other models in general, they allow for style to be preserved on each other. Finally, training on ChilloutMix can kill the photo-realistic style of NED and vice versa.
+
+
+#### Why?
+
+While a thorough explanation on how the model transfers would require much more investigation, I think one way to explain this is by considering the "style vector" pointing to every single model. When training on a merge of various models, these style vectors get corrected or compensated so the style is retained when switching to these closely related models. On the other hand, the source model is unaware of these style vectors. After training on it and transferring to another model the style vectors are added and we indeed recover the style of the new base model.
+
+**A Case Study on AC family**
+
+It is known that BP is trained from AC and NMFSAN is trained from BP, so this provides a good test case on how this affects the final result. According to the above theory, training on upstream model helps more get the style of downstream model, and this seems to be indeed the case.
+
+We get more `fkey` style on nmfsan with the LoHa trained on AC (and not with the LoHa trained on nmfsan)
+![xyz_grid-0024-20230330084608](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0024-20230330084608.jpg)
+
+Same for `shion`
+![xyz_grid-0025-20230330084846](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0025-20230330084846.jpg)
+
+On the other hand, styles trained into LoHas would be less effective if applying from downstream models to upstream model, because what they need to achieve that may be much less than what the upstream model would need. We can also observe in early examples that LoHas trained on AnyLora can give bad result on NAI.
+
+![xyz_grid-0026-20230330085404](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0026-20230330085404.jpg)
+![xyz_grid-0030-20230330090242](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0030-20230330090242.jpg)
+
+
+**Implication for Making Cosplay Images**
+
+To summarize, if you want to have style of some model X, instead of training directly on X it would be better to train on an ancestor of X that does not contain this style.  
+Therefore, if you want to get cosplay of characters, you can do either of the following
+- Train on NED and add a decent amount of photos in regularization set
+- Train on AnyLora and transfer to NED
+
+Lykon did show some successful results by only training with anime images on NED, but I doubt this is really optimal. Actually, he uses again a doll LoRa to reinforce the photo-realistic concept. It may be simper to just do what I suggest above.
+
+
+**Some Myths**
+
+Clearly, the thing that really matters is how the model is made, and not how the model looks like. A model that is versatile in style does not make it a good base model for whatever kind of training. In fact, VBP2-2 has around 300 styles trained in but LoHa trained on top of it does not transfer well to other models.  
+Similarly, two models that produce similar style do not mean they transfer well to each other. Both MFB and Salt-Mix have strong anime screenshot style but a LoHa trained on MFB does not transfer well to Salt-Mix.
+
+
+#### Training Speed
+
+It is also suggested that you train faster on AnyLora. I try to look into this in several ways but I don't see a clear difference.
+
+First, I use the 6000step checkpoints for characters
+![xyz_grid-0007-20230330035309](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0007-20230330035309.jpg)
+![xyz_grid-0012-20230330060311](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0012-20230330060311.jpg)
+
+I do not see big difference between models of the same group.
+In the Tilty example I even see training on NAI, BP, and NMFSAN seems to be faster, but this is only one seed.
+
+I move on to check the 10000step checkpoints for styles
+![xyz_grid-0008-20230330042204](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0008-20230330042204.jpg)
+
+
+Again there is no big difference here.
+
+Finally I use the final checkpoint and set the weight to 0.65 as suggested by Lykon.
+![xyz_grid-0010-20230330051247](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0010-20230330051247.jpg)
+![xyz_grid-0009-20230330044905](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0009-20230330044905.jpg)
+![xyz_grid-0013-20230330062741](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0013-20230330062741.jpg)
+
+The only thing that I observed is that VBP2.2 seems to fit a bit quicker on the styles I test here, but it is probably because it is already trained on them (although under a different triggering mechanism as VBP2.2 is trained with embedding).
+
+
+#### Going further
+
+**What are some good base models for "using" style LoRas**
+
+In the above experiments, it seems that some good candidates that express well the trained styles are
+- VBP2.2
+- Mix-Pro-V3
+- Counterfeit
+
+**What else**
+
+Using different base model can affect things in many other ways, such as the quality or the flexibility of the resulting LoRa. These are however very different to evaluate. I invite you to try the LoHas I trained here to see how they differ in this aspects.
+
+**A style transfer example**
+
+![xyz_grid-0019-20230330082809](https://huggingface.co/alea31415/LyCORIS-experiments/resolve/main/generated_samples_0329/xyz_grid-0019-20230330082809.jpg)
+
+
+
 ### Dataset
 
 Here is the composition of the dataset