Diffusers?
Would the LoRA work via Diffusers too? How are the images fed into the LoRA? Using the image= flag and then an array of images (like the edit models, or fflf), or how?
Would the LoRA work via Diffusers too? How are the images fed into the LoRA? Using the image= flag and then an array of images (like the edit models, or fflf), or how?
Yes, but I don’t expect the quality to be very good if you only load the LoRA.
The important part is not only the LoRA weights, but also the reference/keyframe conditioning workflow. So I would recommend migrating the ComfyUI workflow into a Diffusers script, especially the conditioning and reference video injection part. It’s probably not just a simple image=[...] input like edit models.
In Diffusers you make the image flag an array of images for reference images - but using the LoRA I get this error:
ValueError: Target modules {'attn2.to_k', 'attn1.to_out.0', 'attn1.to_v', 'attn2.to_out.0', 'attn2.to_v', 'attn1.to_k', 'attn1.to_q', 'attn2.to_q'} not found in the base model. Please check the target modules and try again.
In Diffusers you make the image flag an array of images for reference images - but using the LoRA I get this error:
ValueError: Target modules {'attn2.to_k', 'attn1.to_out.0', 'attn1.to_v', 'attn2.to_out.0', 'attn2.to_v', 'attn1.to_k', 'attn1.to_q', 'attn2.to_q'} not found in the base model. Please check the target modules and try again.
Thats because the key name of the lora is not as same as your base model's, you need to modify the name of the keys names manually.