Motion strength setting on new DMD v5 workflow
Hello.
I've noticed a new setting in the initial setup for the scene in the DMD v5 workflow file, namely "Preprocess (Motion Strength)".
What it does exactly? I'm trying to play a bit around it, but I failed to notice what it does in the final videos. I don't happen to see any correlation, even using the same seed, between that setting and the actual motion intensity of the videos.
I tried analyzing the workflow and querying the neural networks. Here's what I found: Motion Strength affects the dynamics of movements. The higher it is, the more roughly the input image is resized, losing some detail and allowing the model to "invent" the movements. This indirectly affects the speed of movements, since the model doesn't "see" the original image properly and more easily invents dynamic movements. I tested it with different condition and motion settings. And movements do indeed become more dynamic with high motion and medium/low condition. But it's not that simple:
The first stage is motion. This is a degradation of the image to further simplify the model, so it can more easily invent movements.
The second stage is conditioning. This is how far the model deviates from the reference image "corrupted" in the first stage. And the higher this value, the less dynamic it will be in order to preserve the reference.
The third stage is prompt. It can also affect the dynamics; if the prompt is "stay still and look at the camera," there probably won't be any dynamics.
But that's not all. I took video samples in the first pass to see the difference, and I did indeed see a "corrupted" video with high motion and a normal one with low motion. And clearly different faces depending on the different conditions. But in the second pass, the face was recognizable in almost all variations. It turned out that in the second pass, the static condition = 1. And the motion is also static and = 30. So, it turns out that the initial parameters don't significantly affect face recognizability in the second pass (in the original TenStrip workflow). So, to make the face change (although in most cases we don't want that), we need to adjust these parameters for the second pass as well.
It's all very cleverly designed.
@TenStrip , did I understand the meaning correctly?
No, I just randomly named that because it kind of controls overall motion allowed. You need to look into how LTX2 conditioning inputs actually work, the blurrier they are the more freedom the model sees. That method went away because sulphur+distilled lora would just change the input and character but all the DMD and reference work I've done lets it work correctly again. And there's a lot of motion at a setting like 40 v.s. 20.