LoRA for improving anatomy (malformed limbs, etc)
Thank you so much for this, and well done on the first CogVideoX-Lora! Great choice with Steamboat Willie; that really put a smile on my face.
I've been following your updates on GitHub quite frequently and have been very excited about the results.
I got the idea to hopefully train a LoRA to improve human anatomy for the model, since hands are often really hard to depict without disfigurement, especially with "too much" movement.
For the hands, I thought about training a LoRA on "Something Something V2," as they have many videos involving human hands. However, I think the captions for these videos might not be very helpful. Thus, I’m equally excited to see your naive video captioning script.
I'm actually running that now with a test batch of 100 videos from Something Something V2, and if the captioning turns out well, I'm thinking about running that LoRA trainer on an H100.
Do you think this idea could be good, or is it unreasonable to think a LoRA like this could help with anatomy?
Hey, thanks for the kind words!
I think what you have in mind might be very reasonable and achievable with finetuning. Captions are very important with CogVideoX as you might have noticed playing around with T2V and I2V. We have an upcoming repository that will be opened soon supporting captioning, dataset preparation, memory efficient training, multiresolution/frame bucketing, etc. at https://github.com/a-r-r-o-w/cogvideox-factory. In fact, the loras I'm uploading are trained from there in under 20 GB on a single 4090 (these are just test loras and not really the best in terms of quality). I think it might be helpful for what you're trying to achieve, so stay tuned!
That sounds very exciting! Thank you for your feedback and I'll be looking forward diving in to your factory.
Hey 👋 The repo is now public! Feel free to report any bugs and issues you face - we plan to continue improving support for video models