Video continuation in longer generations
Has anyone tried to somehow set the last frame of a previous generation as the first one of a new batch to try to get coherence and create longer generations?
With the text2video extension, you can continue from a still. I did a test where I continued from the last frame of each clip and stitched it together, but the results are not super great. There are some other methods in the works to make this better. It will likely require some kind of controlnet to guide the clip continuation and it will probably require a model trained on a longer clip length. Here's the test: https://streamable.com/x03ljm
Wow that's great! Thank you, it's exactly what I was aiming for. I'll look into the python code, I'm trying to get it to work for vid2vid. So the logic would be to take the code part for starting from a still, taking the last processed frame as that still and guide the rest of the generation with the same method as vid2vid, but tbh i'm not sure it's possible or they are two completely different methods. I'll post here if I get anywhere.
Hey! I am trying to run this A1111 A6000 GPU; all I get is streaks. Is there any chance you can share prompts and settings to get it dialed in?
Thanks!
EDIT - Fixed this by not rendering more then 48 frames...
I did actually make an edit to the configuration.json for the model to improve the results with this continuation method. Open it up from 'stable-diffusion-webui\models\ModelScope\t2v' and change line 22 like so: "unet_attn_scales": [0.25, 0.5, 1]
credit goes to za15704080 on discord for this trick!
Hi, is there any way / colab maybe which showcases how to do video inpainting directly with the pipeline?
Only found code for an auto1111 extension in which Im not interested in.
I might be able to implement that myself based on it, but maybe someone has some more experience with this.
Basically, what I want to do is setup the first 12 latent frames with the last 12 of the previous segment (as mentioned earlier here) and also lock them.
But I guess maybe I just have to look into regular inpainting with SD and hack something together..
Is there any way to continue a video given a prompt with a similar context as the original video's prompts and the original video, such that the result is a new video with both the older and newer prompts?