CiaraRowles

/

TemporalNet

like 345

Hi!

So if i'm reading this right, you're trying to re-create this kind of model by using the generated outputs as dataset?

Hi!

Actually, I want to run TemporalNet without WebUI, just using PyTorch codes. So I need to understand the structure of the TemporalNet and how does it work. The above statement " I just made another controlnet model as "temporalnet model". ... And the control weight is 1.5 and 0.7 for ControlNet and TemporalNet respectively" is my understanding. but according to the experiment results, it seems that i didn't figure out how do the TemporalNet work. Please let me try to clarify it:

1. To generate the i-th frame, TemporalNet use the generated (i-1)-th frame as input.
2. The outputs of the TemporalNet are added to the middle and decoder blocks of stable diffusion model along with the corresponding output of the ControlNet. For example, the outputs from the middle block of the TemporalNet and the ControlNet are both added to the middle block of the stable diffusion.
3. The add operation termed in 2 are weighted sum. That is: $z_{sd_{new}} = z_{sd} + 1.5\times z_{controlnet} + 0.7\times z_{temporalnet}$.
4. The parameters of the TemporalNet are loaded from diff_control_sd15_temporalnet_fp16.safetensors and the ControlNet from control_sd15_hed.pth.

Do I misunderstand the structure of the TemporalNet? Could you please correct it for me? Thanks again!