It's hard to tell exactly what the problem is, but I just can't get as good a result as TemporalNet for SD 1.5. I use it for loopback animation e.g. Deforum

With TemporalNet 1.5, when you increased the weight super high, you would enter local minimas with pure saturated cartoony shapes and hard white lines. Never liked that, but you can tell from this visual property that it preserves composition. When I do the same with TemporalNet XL, I cannot get this same artifact with a high ControlNet strength. Instead, the entire image turns kind of glitchy looking. This indicates to me that something about the architecture of TemporalNet has perhaps changed, or it requires update for the SDXL architecture.

TemporalNet 1.5 overall was very efficient, although you would see artifacts by increasing the strength too high in loopback, the amount of change was reduced greatly to where almost no flickering remained. This way I could render frames from scratch with the whole noise schedule start to finish (no img2img) and I could use TemporalNet instead for precise control over consistency. I just had to intermittently spike the strength to a low value so that it could restore full schedule detail & composition with the other ControlNet, avoiding the 'cartoony' look.

I'll try to get concrete examples to show you what I mean but perhaps you already kinda get the jist of it