"hed model"?
You say "Also, I highly recommend you use this in conjunction with the hed model, the settings are already in the script."
Is all that needs changing is the model on the line controlnet2_path = "diffusers/controlnet-canny-sdxl-1.0" ?
I tried changing it to controlnet2_path = "lllyasviel/sd-controlnet-hed" and I now get
File "D:\controlnet-temporalnet-sdxl-1.0\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (154x2048 and 768x320)
Which model should I use and are there any other changes needed?
Using the provided temporalvideo.py with canny gives even more flickering output videos with faded out colors than the original. Is this normal, or is there an updated version of the script?
Here is a source video created with SDXL and the processed result if that helps.
https://softology.pro/Roses.mp4
https://softology.pro/RosesProcessed.mp4
Thanks.
Ah, that's my bad i meant canny for the hed
The flickering is normal, the original temporalnet model did the same, what you need is to run the controlnet with image2image mode at 0.5 denoise, it wasn't out at the time so the provided script is essentially a placeholder that runs the video transformation at a denoise of 1 until that's all going.
OK, thanks. I was hoping hed would be some miracle fix that stopped the flickering. How do I add the 0.5 denoise parameter to the script?
You'll want to modify the script you're using to use the StableDiffusionXLControlNetImg2ImgPipeline merged here https://github.com/huggingface/diffusers/pull/4592
i'm gonna implement it myself on the provided script now knowing it has been released.
OK, if you are going to modify your example code, ie
https://huggingface.co/CiaraRowles/controlnet-temporalnet-sdxl-1.0/blob/main/runtemporalnetxl.py
I will wait for that. I really don't know what I am doing with the ControlNets yet.
I am just really interested to get some of those smooth results like you showed on twitter tagged "TemporalNet + Hed"
https://twitter.com/CiaraRowles1/status/1637486561917906944
Actually looking at that tweet, do you think this is even going to help in my situation? I have movies created with SDXL frames (like the rose movie I linked) that have minor to major flickering. I was interested in seeing if your script could smooth out the flickering, but maybe it is not even suitable for this task?
Oh for that you'll want temporalnet (non xl), this is the one for stable diffusion xl, the tooling for sd xl controlnets are still in progress hence the issues you've run into.
You can find that here https://huggingface.co/CiaraRowles/TemporalNet/tree/main
just drop the dif weights into your automatic1111 UI controlnet weights folder and run it with loopback, like this person did: https://www.youtube.com/watch?v=Er1sz7xVmc8&ab_channel=KoreAI
I am using SDXL though to create that movie and frames. I am trying to get this working outside automatic or other UIs, that is what brought me to your stand alone script.