ByteDance

/

Hyper-SD

like 547

@brandostrong yeah I am also waiting for the 4 step cfg preserved Lora now, but at least the 8 step one came! For animatediff, you could try this currently? It should be better I believe?
https://huggingface.co/ByteDance/AnimateDiff-Lightning

Hi,

I love your work but find it hard to combine with character and subject loras, especially during animatediff due to the low CFG. Without animatediff everything goes fine though. When combining a character lora that is associated with certain colors, the animatediff output tends to be extremely contrasted towards those colors, I'm hoping having higher CFGs will fix this. For example, a character lora with a red outfit will cause the entire video to be shaded red.

Hi, @brandostrong , we have uploaded the CFG-preserved hyper-SD15 LoRA and hyper-SDXL LoRA just now, looking forward to your use and feedback!

@renyuxi Thanks! I've been checking every day.

Hi @renyuxi could you provide a ComfyUI workflow for the 8-step CFG lora?
It's not obvious from the Model Card which sampler name / scheduler must be used to make this work.
My results compared to base SDXL are not great, especially with an added IPadapter like here:
https://storage.googleapis.com/public-assets-xander/Random/remove/8step-cfg-lora.json

Ok, it works without IPadapter, so I guess this LoRa simply isnt compatibel with IPadapter since they both modify the unet weights?

Ok, it works without IPadapter, so I guess this LoRa simply isnt compatibel with IPadapter since they both modify the unet weights?

Hi @aiXander , it is not what we expected, our CFG LoRA should be compatible with the IPAdapters, you can try to use a lower lora weight. If the results are still undesirable, sharing your results and workflow with us will be helpful.

tried with lower LoRa weights but same (bad) results when using IP-adapter, my workflow is attached in the above comment!

Overall I am really impressed with Hyper-SD, it works very well with most straightforward applications. It does get a little more complicated with animatediff and loras however, needing more CFG tuning and weight tuning(5-6 cfg seems to be the sweet spot). There's a pretty noticable quality loss on animatediff, and introduces a lot of artifacting and noise(especially on tiled and upscales). With some tuning it does perform better than LCM and Lightning with animatediff. V3 Adapter is a must.

Hi! I'm finding the output quality of "Hyper-SD15-8steps-CFG-lora.safetensors" to be worse than what I get with the "Hyper-SD15-1step-lora.safetensors" (at 8 steps), is it expected? For the latter I use the TCD scheduler and for the former the DDIM scheduler as in the examples.

Overall I am really impressed with Hyper-SD, it works very well with most straightforward applications. It does get a little more complicated with animatediff and loras however, needing more CFG tuning and weight tuning(5-6 cfg seems to be the sweet spot). There's a pretty noticable quality loss on animatediff, and introduces a lot of artifacting and noise(especially on tiled and upscales). With some tuning it does perform better than LCM and Lightning with animatediff. V3 Adapter is a must.

I want to revise this. It's not so much CFG that's a problem, but basically any LORA that has a trained text encoder is extremely burnt at level 1 strength. A lora with a strength of 1 without hyper-sd(cfg and 1 cfg types) will need 0.3-.5 strength, and has some weird stuff going on with clip sometimes. This is mostly for animatediff, it performs pretty normally with just text2img

Hi, @brandostrong
Some of our users met problems when using upon animatediff here (https://huggingface.co/ByteDance/Hyper-SD/discussions/30).
Could you please share your workflow/script with animatediff?
Thanks a lot!